MongoDB Advisory - A bug in some Linux systems on VMWare leads to namespace file
From:MongoDB [mailto:firstname.lastname@example.org] Sent: Friday, October 10, 2014 7:05 AM To:Maddali, Srinivasacharyulu Subject:MongoDB Advisory: A bug in some Linux systems on VMWare leads to namespace file corruption
We have identified a bug in some Linux systems running on VMware that impacts MongoDB. A request to the Linux kernel to zero a namespace (.ns) file may fail, leaving the previous contents of some disk locations exposed as part of the namespace file. As mongod depends on the initial zeroing of the namespace file by the Linux kernel for correct operation, this effectively corrupts the namespace file, resulting in various MongoDB issues as outlined below.
MongoDB versions 2.6.5 and 2.4.12 contains a workaround for this bug where we manually allocate and zero the namespace file. 2.6.5 (stable) was released on October 8th, 2014 and 2.4.12-rc0 (release candidate) was released on October 9th, 2014, with a stable build of 2.4.12 targeted for approximately one week after the RC. For more information on this issue please see: https://jira.mongodb.org/browse/SERVER-15369.
MongoDB users running Linux on VMWare platforms with the following specification are vulnerable to this issue:
certain 3.x kernels, including Ubuntu 12.x, 14.x, and Fedora 20; and
VMware platform, including VMware ESXi, Fusion, and Workstation; and
emulated SCSI disks using LVM.
To determine whether your Linux system may be affected you can use the following commands:
To find the kernel version you’re running, use: uname -r
If LVM is in use, use “lvs” to list your logical volumes. Compare these volumes with the output of the “mount” command to determine if your MongoDB namespace files are in a volume handled by LVM
Users running any version of MongoDB on an affected platform can be impacted by this issue. The issue occurs when an entry describing a collection or index is first written to a new region of a namespace file, which contains metadata about MongoDB namespaces (collections or indexes) for a particular database. If this occurs while in a configuration described above, that region of the namespace file may be improperly zeroed by the Linux kernel, and the non-zero left-over data in the namespace file is interpreted by mongod as corrupted namespace data; since the leftover data is unpredictable, symptoms can vary.
Symptoms that have been observed include:
failures of various operations that require enumerating namespaces, such as db.stats(), db.repairDatabase(), db.dropDatabase(), and others
operations such as the above that don’t fail may report nonexistent database names
failure of mongod to start
These failures may be accompanied by logged error messages such as:
Additional symptoms may be possible depending on the exact nature of the left-over data in the namespace file.
MongoDB users prior to 2.6 may have namespace file damage caused by this issue and not notice symptoms until an attempt to upgrade to 2.6, as 2.6 introduces additional checks that expose the issue.
The occurrence of namespace file corruption is dependent on the specific database, collection, and index names that are stored in the namespace file, but if a pattern of names that triggers the problem is present, the problem is likely to be repeatable. For example, the corruption may occur every time a given data set is loaded. For this reason the problem may also affect multiple members of a replica set.
No verified external workarounds exist for this issue. Contact MongoDB support if you need the latest information about potential external workarounds. To avoid namespace file corruption, users must either apply a vendor-supplied patch to their Linux kernel, or in the meantime upgrade to MongoDB 2.4.12 or 2.6.5.
Note that this avoids future namespace file corruption, but does not repair existing damage. In addition, a namespace file may show no symptoms of damage but still be vulnerable to future corruption if it was created by a version of MongoDB prior to this workaround, and an updated kernel with a fix for the issue has not been installed. Steps to repair existing damage and prevent future damage to existing namespace files can vary depending on circumstances, so if you suspect you have encountered this issue or are may be vulnerable to it, please raise a support case as described above for assistance.