Navigation

Upgrade MongoDB to 2.4

On this page

In the general case, the upgrade from MongoDB 2.2 to 2.4 is a binary-compatible “drop-in” upgrade: shut down the mongod instances and replace them with mongod instances running 2.4. However, before you attempt any upgrade please familiarize yourself with the content of this document, particularly the procedure for upgrading sharded clusters and the considerations for reverting to 2.2 after running 2.4.

Upgrade Recommendations and Checklist

When upgrading, consider the following:

  • For all deployments using authentication, upgrade the drivers (i.e. client libraries), before upgrading the mongod instance or instances.

  • To upgrade to 2.4 sharded clusters must upgrade following the meta-data upgrade procedure.

  • If you’re using 2.2.0 and running with authorization enabled, you will need to upgrade first to 2.2.1 and then upgrade to 2.4. See Rolling Upgrade Limitation for 2.2.0 Deployments Running with auth Enabled.

  • If you have system.users documents (i.e. for authorization) that you created before 2.4 you must ensure that there are no duplicate values for the user field in the system.users collection in any database. If you do have documents with duplicate user fields, you must remove them before upgrading.

    See Security Enhancements for more information.

Upgrade Standalone mongod Instance to MongoDB 2.4

  1. Download binaries of the latest release in the 2.4 series from the MongoDB Download Page. See Install MongoDB for more information.
  2. Shutdown your mongod instance. Replace the existing binary with the 2.4 mongod binary and restart mongod.

Upgrade a Replica Set from MongoDB 2.2 to MongoDB 2.4

You can upgrade to 2.4 by performing a “rolling” upgrade of the set by upgrading the members individually while the other members are available to minimize downtime. Use the following procedure:

  1. Upgrade the secondary members of the set one at a time by shutting down the mongod and replacing the 2.2 binary with the 2.4 binary. After upgrading a mongod instance, wait for the member to recover to SECONDARY state before upgrading the next instance. To check the member’s state, issue rs.status() in the mongo shell.

  2. Use the mongo shell method rs.stepDown() to step down the primary to allow the normal failover procedure. rs.stepDown() expedites the failover procedure and is preferable to shutting down the primary directly.

    Once the primary has stepped down and another member has assumed PRIMARY state, as observed in the output of rs.status(), shut down the previous primary and replace mongod binary with the 2.4 binary and start the new process.

    Note

    Replica set failover is not instant but will render the set unavailable to read or accept writes until the failover process completes. Typically this takes 10 seconds or more. You may wish to plan the upgrade during a predefined maintenance window.

Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4

Important

Only upgrade sharded clusters to 2.4 if all members of the cluster are currently running instances of 2.2. The only supported upgrade path for sharded clusters running 2.0 is via 2.2.

Overview

Upgrading a sharded cluster from MongoDB version 2.2 to 2.4 (or 2.3) requires that you run a 2.4 mongos with the --upgrade option, described in this procedure. The upgrade process does not require downtime.

The upgrade to MongoDB 2.4 adds epochs to the meta-data for all collections and chunks in the existing cluster. MongoDB 2.2 processes are capable of handling epochs, even though 2.2 did not require them. This procedure applies only to upgrades from version 2.2. Earlier versions of MongoDB do not correctly handle epochs. See Cluster Meta-data Upgrade for more information.

After completing the meta-data upgrade you can fully upgrade the components of the cluster. With the balancer disabled:

  • Upgrade all mongos instances in the cluster.
  • Upgrade all 3 mongod config server instances.
  • Upgrade the mongod instances for each shard, one at a time.

See Upgrade Sharded Cluster Components for more information.

Cluster Meta-data Upgrade

Considerations

Beware of the following properties of the cluster upgrade process:

  • Before you start the upgrade, ensure that the amount of free space on the filesystem for the config database is at least 4 to 5 times the amount of space currently used by the config database data files.

    Additionally, ensure that all indexes in the config database are {v:1} indexes. If a critical index is a {v:0} index, chunk splits can fail due to known issues with the {v:0} format. {v:0} indexes are present on clusters created with MongoDB 2.0 or earlier.

    The duration of the metadata upgrade depends on the network latency between the node performing the upgrade and the three config servers. Ensure low latency between the upgrade process and the config servers.

  • While the upgrade is in progress, you cannot make changes to the collection meta-data. For example, during the upgrade, do not perform:

  • Once you upgrade to 2.4 and complete the upgrade procedure do not use 2.0 mongod and mongos processes in your cluster. 2.0 process may re-introduce old meta-data formats into cluster meta-data.

The upgraded config database will require more storage space than before, to make backup and working copies of the config.chunks and config.collections collections. As always, if storage requirements increase, the mongod might need to pre-allocate additional data files. See How can I get information on the storage use of a database? for more information.

Meta-data Upgrade Procedure

Changes to the meta-data format for sharded clusters, stored in the config database, require a special meta-data upgrade procedure when moving to 2.4.

Do not perform operations that modify meta-data while performing this procedure. See Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4 for examples of prohibited operations.

  1. Before you start the upgrade, ensure that the amount of free space on the filesystem for the config database is at least 4 to 5 times the amount of space currently used by the config database data files.

    Additionally, ensure that all indexes in the config database are {v:1} indexes. If a critical index is a {v:0} index, chunk splits can fail due to known issues with the {v:0} format. {v:0} indexes are present on clusters created with MongoDB 2.0 or earlier.

    The duration of the metadata upgrade depends on the network latency between the node performing the upgrade and the three config servers. Ensure low latency between the upgrade process and the config servers.

    To check the version of your indexes, use db.collection.getIndexes().

    If any index on the config database is {v:0}, you should rebuild those indexes by connecting to the mongos and either: rebuild all indexes using the db.collection.reIndex() method, or drop and rebuild specific indexes using db.collection.dropIndex() and then db.collection.ensureIndex(). If you need to upgrade the _id index to {v:1} use db.collection.reIndex().

    You may have {v:0} indexes on other databases in the cluster.

  2. Turn off the balancer in the sharded cluster, as described in Disable the Balancer.

    Optional

    For additional security during the upgrade, you can make a backup of the config database using mongodump or other backup tools.

  3. Ensure there are no version 2.0 mongod or mongos processes still active in the sharded cluster. The automated upgrade process checks for 2.0 processes, but network availability can prevent a definitive check. Wait 5 minutes after stopping or upgrading version 2.0 mongos processes to confirm that none are still active.

  4. Start a single 2.4 mongos process with configDB pointing to the sharded cluster’s config servers and with the --upgrade option. The upgrade process happens before the process becomes a daemon (i.e. before --fork.)

    You can upgrade an existing mongos instance to 2.4 or you can start a new mongos instance that can reach all config servers if you need to avoid reconfiguring a production mongos.

    Start the mongos with a command that resembles the following:

    copy 
    mongos --configdb <config servers> --upgrade

    Without the --upgrade option 2.4 mongos processes will fail to start until the upgrade process is complete.

    The upgrade will prevent any chunk moves or splits from occurring during the upgrade process. If there are very many sharded collections or there are stale locks held by other failed processes, acquiring the locks for all collections can take seconds or minutes. See the log for progress updates.

  5. When the mongos process starts successfully, the upgrade is complete. If the mongos process fails to start, check the log for more information.

    If the mongos terminates or loses its connection to the config servers during the upgrade, you may always safely retry the upgrade.

    However, if the upgrade failed during the short critical section, the mongos will exit and report that the upgrade will require manual intervention. To continue the upgrade process, you must follow the Resync after an Interruption of the Critical Section procedure.

    Optional

    If the mongos logs show the upgrade waiting for the upgrade lock, a previous upgrade process may still be active or may have ended abnormally. After 15 minutes of no remote activity mongos will force the upgrade lock. If you can verify that there are no running upgrade processes, you may connect to a 2.2 mongos process and force the lock manually:

    copy 
    mongo <mongos.example.net>
    copy 
    db.getMongo().getCollection("config.locks").findOne({ _id : "configUpgrade" })

    If the process specified in the process field of this document is verifiably offline, run the following operation to force the lock.

    copy 
    db.getMongo().getCollection("config.locks").update({ _id : "configUpgrade" }, { $set : { state : 0 } })

    It is always more safe to wait for the mongos to verify that the lock is inactive, if you have any doubts about the activity of another upgrade operation. In addition to the configUpgrade, the mongos may need to wait for specific collection locks. Do not force the specific collection locks.

  6. Upgrade and restart other mongos processes in the sharded cluster, without the --upgrade option.

    See Upgrade Sharded Cluster Components for more information.

  7. Re-enable the balancer. You can now perform operations that modify cluster meta-data.

Once you have upgraded, do not introduce version 2.0 MongoDB processes into the sharded cluster. This can reintroduce old meta-data formats into the config servers. The meta-data change made by this upgrade process will help prevent errors caused by cross-version incompatibilities in future versions of MongoDB.

Resync after an Interruption of the Critical Section

During the short critical section of the upgrade that applies changes to the meta-data, it is unlikely but possible that a network interruption can prevent all three config servers from verifying or modifying data. If this occurs, the config servers must be re-synced, and there may be problems starting new mongos processes. The sharded cluster will remain accessible, but avoid all cluster meta-data changes until you resync the config servers. Operations that change meta-data include: adding shards, dropping databases, and dropping collections.

Note

Only perform the following procedure if something (e.g. network, power, etc.) interrupts the upgrade process during the short critical section of the upgrade. Remember, you may always safely attempt the meta data upgrade procedure.

To resync the config servers:

  1. Turn off the balancer in the sharded cluster and stop all meta-data operations. If you are in the middle of an upgrade process (Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4), you have already disabled the balancer.

  2. Shut down two of the three config servers, preferably the last two listed in the configDB string. For example, if your configDB string is configA:27019,configB:27019,configC:27019, shut down configB and configC. Shutting down the last two config servers ensures that most mongos instances will have uninterrupted access to cluster meta-data.

  3. mongodump the data files of the active config server (configA).

  4. Move the data files of the deactivated config servers (configB and configC) to a backup location.

  5. Create new, empty data directories.

  6. Restart the disabled config servers with --dbpath pointing to the now-empty data directory and --port pointing to an alternate port (e.g. 27020).

  7. Use mongorestore to repopulate the data files on the disabled documents from the active config server (configA) to the restarted config servers on the new port (configB:27020,configC:27020). These config servers are now re-synced.

  8. Restart the restored config servers on the old port, resetting the port back to the old settings (configB:27019 and configC:27019).

  9. In some cases connection pooling may cause spurious failures, as the mongos disables old connections only after attempted use. 2.4 fixes this problem, but to avoid this issue in version 2.2, you can restart all mongos instances (one-by-one, to avoid downtime) and use the rs.stepDown() method before restarting each of the shard replica set primaries.

  10. The sharded cluster is now fully resynced; however before you attempt the upgrade process again, you must manually reset the upgrade state using a version 2.2 mongos. Begin by connecting to the 2.2 mongos with the mongo shell:

    copy 
    mongo <mongos.example.net>

    Then, use the following operation to reset the upgrade process:

    copy 
    db.getMongo().getCollection("config.version").update({ _id : 1 }, { $unset : { upgradeState : 1 } })

  11. Finally retry the upgrade process, as in Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4.

Upgrade Sharded Cluster Components

After you have successfully completed the meta-data upgrade process described in Meta-data Upgrade Procedure, and the 2.4 mongos instance starts, you can upgrade the other processes in your MongoDB deployment.

While the balancer is still disabled, upgrade the components of your sharded cluster in the following order:

  • Upgrade all mongos instances in the cluster, in any order.
  • Upgrade all 3 mongod config server instances, upgrading the first system in the mongos --configdb argument last.
  • Upgrade each shard, one at a time, upgrading the mongod secondaries before running replSetStepDown and upgrading the primary of each shard.

When this process is complete, you can now re-enable the balancer.

Rolling Upgrade Limitation for 2.2.0 Deployments Running with auth Enabled

MongoDB cannot support deployments that mix 2.2.0 and 2.4.0, or greater, components. MongoDB version 2.2.1 and later processes can exist in mixed deployments with 2.4-series processes. Therefore you cannot perform a rolling upgrade from MongoDB 2.2.0 to MongoDB 2.4.0. To upgrade a cluster with 2.2.0 components, use one of the following procedures.

  1. Perform a rolling upgrade of all 2.2.0 processes to the latest 2.2-series release (e.g. 2.2.3) so that there are no processes in the deployment that predate 2.2.1. When there are no 2.2.0 processes in the deployment, perform a rolling upgrade to 2.4.0.
  2. Stop all processes in the cluster. Upgrade all processes to a 2.4-series release of MongoDB, and start all processes at the same time.

Upgrade from 2.3 to 2.4

If you used a mongod from the 2.3 or 2.4-rc (release candidate) series, you can safely transition these databases to 2.4.0 or later; however, if you created 2dsphere or text indexes using a mongod before v2.4-rc2, you will need to rebuild these indexes. For example:

copy 
db.records.dropIndex( { loc: "2dsphere" } )
db.records.dropIndex( "records_text" )

db.records.ensureIndex( { loc: "2dsphere" } )
db.records.ensureIndex( { records: "text" } )

Downgrade MongoDB from 2.4 to Previous Versions

For some cases the on-disk format of data files used by 2.4 and 2.2 mongod is compatible, and you can upgrade and downgrade if needed. However, several new features in 2.4 are incompatible with previous versions:

  • 2dsphere indexes are incompatible with 2.2 and earlier mongod instances.
  • text indexes are incompatible with 2.2 and earlier mongod instances.
  • using a hashed index as a shard key are incompatible with 2.2 and earlier mongos instances.
  • hashed indexes are incompatible with 2.0 and earlier mongod instances.

Important

Collections sharded using hashed shard keys, should not use 2.2 mongod instances, which cannot correctly support cluster operations for these collections.

If you completed the meta-data upgrade for a sharded cluster, you can safely downgrade to 2.2 MongoDB processes. Do not use 2.0 processes after completing the upgrade procedure.

Note

In sharded clusters, once you have completed the meta-data upgrade procedure, you cannot use 2.0 mongod or mongos instances in the same cluster.

If you complete the meta-data upgrade, you can safely downgrade components in any order. When upgrade again, always upgrade mongos instances before mongod instances.

Do not create 2dsphere or text indexes in a cluster that has 2.2 components.

Considerations and Compatibility

If you upgrade to MongoDB 2.4, and then need to run MongoDB 2.2 with the same data files, consider the following limitations.

  • If you use a hashed index as the shard key index, which is only possible under 2.4 you will not be able to query data in this sharded collection. Furthermore, a 2.2 mongos cannot properly route an insert operation for a collections sharded using a hashed index for the shard key index: any data that you insert using a 2.2 mongos, will not arrive on the correct shard and will not be reachable by future queries.
  • If you never create an 2dsphere or text index, you can move between a 2.4 and 2.2 mongod for a given data set; however, after you create the first 2dsphere or text index with a 2.4 mongod you will need to run a 2.2 mongod with the --upgrade option and drop any 2dsphere or text index.

Upgrade and Downgrade Procedures

Basic Downgrade and Upgrade

Except as described below, moving between 2.2 and 2.4 is a drop-in replacement:

  • stop the existing mongod, using the --shutdown option as follows:

    copy 
    mongod --dbpath /var/mongod/data --shutdown

    Replace /var/mongod/data with your MongoDB dbPath.

  • start the new mongod processes with the same dbPath setting, for example:

    copy 
    mongod --dbpath /var/mongod/data

    Replace /var/mongod/data with your MongoDB dbPath.

Downgrade to 2.2 After Creating a 2dsphere or text Index

If you have created 2dsphere or text indexes while running a 2.4 mongod instance, you can downgrade at any time, by starting the 2.2 mongod with the --upgrade option as follows:

copy 
mongod --dbpath /var/mongod/data/ --upgrade

Then, you will need to drop any existing 2dsphere or text indexes using db.collection.dropIndex(), for example:

copy 
db.records.dropIndex( { loc: "2dsphere" } )
db.records.dropIndex( "records_text" )

Warning

--upgrade will run repairDatabase on any database where you have created a 2dsphere or text index, which will rebuild all indexes.

Troubleshooting Upgrade/Downgrade Operations

If you do not use --upgrade, when you attempt to start a 2.2 mongod and you have created a 2dsphere or text index, mongod will return the following message:

copy 
'need to upgrade database index_plugin_upgrade with pdfile version 4.6, new version: 4.5 Not upgrading, exiting'

While running 2.4, to check the data file version of a MongoDB database, use the following operation in the shell:

copy 
db.getSiblingDB('<databasename>').stats().dataFileVersion

The major data file [1] version for both 2.2 and 2.4 is 4, the minor data file version for 2.2 is 5 and the minor data file version for 2.4 is 6 after you create a 2dsphere or text index.

[1]The data file version (i.e. pdfile version) is independent and unrelated to the release version of MongoDB.

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.