Upgrade a Sharded Cluster to 4.0¶

On this page

Upgrade Recommendations and Checklists
Prerequisites
Download 4.0 Binaries
Upgrade Process
Additional Upgrade Procedures

Important

MongoDB 4.0 may lose data during unclean shutdowns on macOS 10.12.x and 10.13.x.

For details, see WT-4018.

Important

Before you attempt any upgrade, please familiarize yourself with the content of this document.

If you need guidance on upgrading to 4.0, MongoDB offers major version upgrade services to help ensure a smooth transition without interruption to your MongoDB application.

Upgrade Recommendations and Checklists¶

When upgrading, consider the following:

Upgrade Version Path¶

To upgrade an existing MongoDB deployment to 4.0, you must be running a 3.6-series release.

To upgrade from a version earlier than the 3.6-series, you must successively upgrade major releases until you have upgraded to 3.6-series. For example, if you are running a 3.4-series, you must upgrade first to 3.6 before you can upgrade to 4.0.

Preparedness¶

Before beginning your upgrade, see the Compatibility Changes in MongoDB 4.0 document to ensure that your applications and deployments are compatible with MongoDB 4.0. Resolve the incompatibilities in your deployment before starting the upgrade.

Before upgrading MongoDB, always test your application in a staging environment before deploying the upgrade to your production environment.

Downgrade Consideration¶

Once upgraded to 4.0, if you need to downgrade, we recommend downgrading to the latest patch release of 3.6.

Default Bind to Localhost¶

The following procedure includes the command-line option --bind_ip or the configuration option net.bindIp when restarting the replica set.

Starting in MongoDB 3.6, the options must be specified when the replica set members are run on different hosts or if remote clients connect to the deployment. Omit if all members are run on the same host and all clients are local to the host.

Warning

Before binding to a non-localhost (e.g. publicly accessible) IP address, ensure you have secured your cluster from unauthorized access. For a complete list of security recommendations, see Security Checklist. At minimum, consider enabling authentication and hardening network infrastructure.

Read Concern Majority (3-Member Primary-Secondary-Arbiter Architecture)¶

Starting in MongoDB 3.6, MongoDB enables support for "majority" read concern by default.

For MongoDB 4.0.3+ (and 3.6.1+), you can disable read concern "majority" to prevent the storage cache pressure from immobilizing a deployment with a three-member replica set with a primary-secondary-arbiter (PSA) architecture or a sharded cluster with a three-member PSA shards.

Note

Disabling "majority" read concern disables support for Change Streams for MongoDB 4.0 and earlier. For MongoDB 4.2+, disabling read concern "majority" has no effect on change streams availability.

Disabling "majority" does not affect multi-document transactions; i.e. you can specify read concern "majority" for transactions even if read concern "majority" is disabled.

For more information, see Disable Read Concern Majority.

Change Streams Resume Tokens¶

MongoDB 4.0 introduces new hex-encoded string change stream resume tokens:

The resume token _data type depends on the MongoDB versions and, in some cases, the feature compatibility version (fcv) at the time of the change stream’s opening/resumption (i.e. a change in fcv value does not affect the resume tokens for already opened change streams):

MongoDB Version	Feature Compatibility Version	Resume Token `_data` Type
MongoDB 4.0.7 and later	“4.0” or “3.6”	Hex-encoded string (`v1`)
MongoDB 4.0.6 and earlier	“4.0”	Hex-encoded string (`v0`)
MongoDB 4.0.6 and earlier	“3.6”	BinData
MongoDB 3.6	“3.6”	BinData

When upgrading from MongoDB 3.6 to MongoDB 4.0.7

When upgrading from MongoDB 3.6 to MongoDB 4.0.7 or later, the upgraded sharded cluster members continue to produce v0 token until the first mongos instance is upgraded. If a client tries to resume change streams using the new v1 resume token when connected to another mongos that has not been updated (i.e. only accepts BinData resume tokens), the resume operation will fail. In such cases, the client must wait for the upgrade to complete before resuming change streams.

After upgrading, if you later decide to downgrade to MongoDB 3.6, to resume a change stream, clients can use a pre-upgrade resume token (if available) on the 3.6 deployment. Otherwise, clients will need to start a new change stream.

Prerequisites¶

All Members Version¶

To upgrade a sharded cluster to 4.0, all members of the cluster must be at least version 3.6. The upgrade process checks all components of the cluster and will produce warnings if any component is running version earlier than 3.6.

Replica Set Member State¶

For shards and config servers, ensure that no replica set member is in ROLLBACK or RECOVERING state.

Remove Support for `MONGODB-CR`¶

Starting in version 4.0, MongoDB removes support for the deprecated MongoDB Challenge-Response (MONGODB-CR) authentication mechanism.

If your deployment has user credentials stored in MONGODB-CR schema, you must upgrade to Salted Challenge Response Authentication Mechanism (SCRAM) before you upgrade to version 4.0. For information on upgrading to SCRAM, see Upgrade to SCRAM.

Remove `pv0` for Replica Sets¶

Starting in version 4.0, MongoDB removes the deprecated replica set protocol version 0 pv0.

Before upgrading to MongoDB 4.0, you must upgrade to pv1.

To upgrade to pv1, connect a mongo shell to the replica set primary and perform the following sequence of operations:

copy

cfg = rs.conf();
cfg.protocolVersion=1;
rs.reconfig(cfg);

To reduce the likelihood of w:1 rollbacks, you can also reconfigure the replica set to a higher settings.catchUpTimeoutMillis setting.

For more information on pv1, see Replica Set Protocol Version.

Remove Master-Slave Replication¶

MongoDB 4.0 removes support for the deprecated master-slave replication. Before you can upgrade to MongoDB 4.0, if your deployment uses master-slave replication, you must upgrade to a replica set.

To convert from master-slave replication to a replica set, see Convert a Master-Slave Deployment to a Replica Set.

Remove Support for `$isolated`¶

MongoDB drops support for the $isolated operator. If you have an existing partial index that includes the $isolated operator or a view that includes a $isolated operator, recreate the index or view without the operator in the definition before upgrading.

Feature Compatibility Version¶

The 3.6 sharded cluster must have featureCompatibilityVersion set to 3.6.

To ensure that all members of the sharded cluster have featureCompatibilityVersion set to 3.6, connect to each shard replica set member and each config server replica set member and check the featureCompatibilityVersion:

Tip

For a sharded cluster that has access control enabled, to run the following command against a shard replica set member, you must connect to the member as a shard local user.

copy

db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

All members should return a result that includes "featureCompatibilityVersion" : { "version" : "3.6" }.

To set or update featureCompatibilityVersion, run the following command on the mongos:

copy

db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

For more information, see setFeatureCompatibilityVersion.

Back up the `config` Database¶

Optional but Recommended. As a precaution, take a backup of the config database before upgrading the sharded cluster.

Download 4.0 Binaries¶

Use Package Manager¶

If you installed MongoDB from the MongoDB apt, yum, dnf, or zypper repositories, you should upgrade to 4.0 using your package manager.

Follow the appropriate 4.0 installation instructions for your Linux system. This will involve adding a repository for the new release, then performing the actual upgrade process.

Download 4.0 Binaries Manually¶

If you have not installed MongoDB using a package manager, you can manually download the MongoDB binaries from the MongoDB Download Center.

See 4.0 installation instructions for more information.

Upgrade Process¶

1

Disable the Balancer.¶

Connect a mongo shell to a mongos instance in the sharded cluster, and run sh.stopBalancer() to disable the balancer:

copy

sh.stopBalancer()

Note

If a migration is in progress, the system will complete the in-progress migration before stopping the balancer. You can run sh.isBalancerRunning() to check the balancer’s current state.

To verify that the balancer is disabled, run sh.getBalancerState(), which returns false if the balancer is disabled:

copy

sh.getBalancerState()

For more information on disabling the balancer, see Disable the Balancer.

2

Upgrade the config servers.¶

Upgrade the secondary members of the replica set one at a time:
1. Shut down the secondary mongod instance and replace the 3.6 binary with the 4.0 binary.
2. Start the 4.0 binary with the --configsvr, --replSet, and --port. Include any other options as used by the deployment.
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.
  copy
  mongod --configsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>
  If using a configuration file, update the file to specify sharding.clusterRole: configsvr, replication.replSetName, net.port, and net.bindIp, then start the 4.0 binary:
  copy
  sharding: clusterRole: configsvr replication: replSetName: <string> net: port: <port> bindIp: localhost,<hostname(s)|ip address(es)> storage: dbpath: <path>
  Include any other settings as appropriate for your deployment.
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.
3. Wait for the member to recover to SECONDARY state before upgrading the next secondary member. To check the member’s state, issue rs.status() in the mongo shell.
  
  Repeat for each secondary member.
Step down the replica set primary.
1. Connect a mongo shell to the primary and use rs.stepDown() to step down the primary and force an election of a new primary:
  copy
  rs.stepDown()
2. When rs.status() shows that the primary has stepped down and another member has assumed PRIMARY state, shut down the stepped-down primary and replace the mongod binary with the 4.0 binary.
3. Start the 4.0 binary with the --configsvr, --replSet, --port, and --bind_ip options. Include any optional command line options used by the previous deployment:
  copy
  mongod --configsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.
  If using a configuration file, update the file to specify sharding.clusterRole: configsvr, replication.replSetName, net.port, and net.bindIp, then start the 4.0 binary:
  copy
  sharding: clusterRole: configsvr replication: replSetName: <string> net: port: <port> bindIp: localhost,<hostname(s)|ip address(es)> storage: dbpath: <path>
  Include any other configuration as appropriate for your deployment.
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.

3

Upgrade the shards.¶

Upgrade the shards one at a time.

For each shard replica set:

Upgrade the secondary members of the replica set one at a time:
1. Shut down the mongod instance and replace the 3.6 binary with the 4.0 binary.
2. Start the 4.0 binary with the --shardsvr, --replSet, --port, and --bind_ip options. Include any additional command line options as appropriate for your deployment:
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.
  copy
  mongod --shardsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>
  If using a configuration file, update the file to include sharding.clusterRole: shardsvr, replication.replSetName, net.port, and net.bindIp, then start the 4.0 binary:
  copy
  sharding: clusterRole: shardsvr replication: replSetName: <string> net: port: <port> bindIp: localhost,<hostname(s)|ip address(es)> storage: dbpath: <path>
  Include any other configuration as appropriate for your deployment.
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.
3. Wait for the member to recover to SECONDARY state before upgrading the next secondary member. To check the member’s state, you can issue rs.status() in the mongo shell.
  
  Repeat for each secondary member.
Step down the replica set primary.

Connect a mongo shell to the primary and use rs.stepDown() to step down the primary and force an election of a new primary:
copy
```
rs.stepDown()
```
When rs.status() shows that the primary has stepped down and another member has assumed PRIMARY state, upgrade the stepped-down primary:
1. Shut down the stepped-down primary and replace the mongod binary with the 4.0 binary.
2. Start the 4.0 binary with the --shardsvr, --replSet, --port, and --bind_ip options. Include any additional command line options as appropriate for your deployment:
  copy
  mongod --shardsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.
  If using a configuration file, update the file to specify sharding.clusterRole: shardsvr, replication.replSetName, net.port, and net.bindIp, then start the 4.0 binary:
  copy
  sharding: clusterRole: shardsvr replication: replSetName: <string> net: port: <port> bindIp: localhost,<hostname(s)|ip address(es)> storage: dbpath: <path>
  Include any other configuration as appropriate for your deployment.
  Note
  - Starting in MongoDB 4.0, you cannot specify --nojournal option or storage.journal.enabled: false for replica set members that use the WiredTiger storage engine.
  - Starting in MongoDB 4.0, you cannot use --noIndexBuildRetry or storage.indexBuildRetry for a mongod instance that is part of a replica set.

4

Upgrade the `mongos` instances.¶

Replace each mongos instance with the 4.0 binary and restart. Include any other configuration as appropriate for your deployment.

Note

The --bind_ip option must be specified when the sharded cluster members are run on different hosts or if remote clients connect to the sharded cluster. For more information, see Localhost Binding Compatibility Changes.

copy

mongos --configdb csReplSet/<rsconfigsver1:port1>,<rsconfigsver2:port2>,<rsconfigsver3:port3> --bind_ip localhost,<hostname(s)|ip address(es)>

5

Re-enable the balancer.¶

Using a 4.0 mongo shell, connect to a mongos in the cluster and run sh.setBalancerState() to re-enable the balancer:

copy

sh.setBalancerState(true)

The 3.6 and earlier mongo shell is not compatible with 4.0 clusters.

For more information about re-enabling the balancer, see Enable the Balancer.

6

Enable backwards-incompatible 4.0 features.¶

At this point, you can run the 4.0 binaries without the 4.0 features that are incompatible with 3.6.

To enable these 4.0 features, set the feature compatibility version (FCV) to 4.0.

Tip

Enabling these backwards-incompatible features can complicate the downgrade process since you must remove any persisted backwards-incompatible features before you downgrade.

It is recommended that after upgrading, you allow your deployment to run without enabling these features for a burn-in period to ensure the likelihood of downgrade is minimal. When you are confident that the likelihood of downgrade is minimal, enable these features.

On a mongos instance, run the setFeatureCompatibilityVersion command in the admin database:

copy

db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } )

This command must perform writes to an internal system collection. If for any reason the command does not complete successfully, you can safely retry the command on the mongos as the operation is idempotent.

7

Restart `mongos` instances.¶

After changing the featureCompatibilityVersion, all mongos instances need to be restarted to pick up the changes in the causal consistency behavior.

Additional Upgrade Procedures¶

To upgrade a standalone, see Upgrade a Standalone to 4.0.
To upgrade a replica set, see Upgrade a Replica Set to 4.0.

← Upgrade a Replica Set to 4.0 Downgrade 4.0 Standalone to 3.6 →

Upgrade a Sharded Cluster to 4.0¶

Upgrade Recommendations and Checklists¶

Upgrade Version Path¶

Preparedness¶

Downgrade Consideration¶

Default Bind to Localhost¶

Read Concern Majority (3-Member Primary-Secondary-Arbiter Architecture)¶

Change Streams Resume Tokens¶

Prerequisites¶

All Members Version¶

Replica Set Member State¶

Remove Support for MONGODB-CR¶

Remove pv0 for Replica Sets¶

Remove Master-Slave Replication¶

Remove Support for $isolated¶

Feature Compatibility Version¶

Back up the config Database¶

Download 4.0 Binaries¶

Use Package Manager¶

Download 4.0 Binaries Manually¶

Upgrade Process¶

Disable the Balancer.¶

Upgrade the config servers.¶

Upgrade the shards.¶

Upgrade the mongos instances.¶

Re-enable the balancer.¶

Enable backwards-incompatible 4.0 features.¶

Restart mongos instances.¶

Additional Upgrade Procedures¶

Remove Support for `MONGODB-CR`¶

Remove `pv0` for Replica Sets¶

Remove Support for `$isolated`¶

Back up the `config` Database¶

Upgrade the `mongos` instances.¶

Restart `mongos` instances.¶