Change Streams¶
On this page
New in version 3.6.
Change streams allow applications to access real-time data changes without the complexity and risk of tailing the oplog. Applications can use change streams to subscribe to all data changes on a single collection, a database, or an entire deployment, and immediately react to them. Because change streams use the aggregation framework, applications can also filter for specific changes or transform the notifications at will.
Availability¶
Change streams are available for replica sets and sharded clusters:
Storage Engine.
The replica sets and sharded clusters must use the WiredTiger storage engine. Change streams can also be used on deployments that employ MongoDB’s encryption-at-rest feature.
Replica Set Protocol Version.
The replica sets and sharded clusters must use replica set protocol version 1 (
pv1
).Read Concern “majority” Enablement.
Starting in MongoDB 4.2, change streams are available regardless of the
"majority"
read concern support; that is, read concernmajority
support can be either enabled (default) or disabled to use change streams.In MongoDB 4.0 and earlier, change streams are available only if
"majority"
read concern support is enabled (default).
Watch Collection/Database/Deployment¶
You can open change streams against:
Target | Description |
---|---|
A collection | You can open a change stream cursor for a single collection
(except The examples on this page use the MongoDB drivers to open and
work with a change stream cursor for a single collection. See
also the |
A database | Starting in MongoDB 4.0, you can open a change stream cursor for
a single database (excluding For the MongoDB driver method, refer to your driver
documentation. See also the |
A deployment | Starting in MongoDB 4.0, you can open a change stream cursor for
a deployment (either a replica set or a sharded cluster) to
watch for changes to all non-system collections across all
databases except for For the MongoDB driver method, refer to your driver
documentation. See also the |
Change Stream Examples
The examples on this page use the MongoDB drivers to illustrate how to open a change stream cursor for a collection and work with the change stream cursor.
Open A Change Stream¶
To open a change stream:
- For a replica set, you can issue the open change stream operation from any of the data-bearing members.
- For a sharded cluster, you must issue the open change stream
operation from the
mongos
.
The following example opens a change stream for a collection and iterates over the cursor to retrieve the change stream documents. [1]
- Python
- Java (Sync)
- Node.js
- PHP
- Motor
- Other
The Python examples below assume that you have connected to a MongoDB replica set and have accessed a database that contains an inventory
collection.
The Java examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The Node.js examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The following example uses stream to process the change events.
Alternatively, you can also use iterator to process the change events:
The examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The C examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The C# examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The Go examples below assume that you have connected to a MongoDB replica set and have accessed a database
that contains an inventory
collection.
The Swift (Sync) examples below assume that you have
connected to a MongoDB replica set and have accessed a
database
that contains an inventory
collection.
The Swift (Async) examples below assume that you have
connected to a MongoDB replica set and have accessed a
database
that contains an inventory
collection.
To retrieve the data change event from the cursor, iterate the change stream cursor. For information on the change stream event, see Change Events.
While the connection to the MongoDB deployment remains open, the cursor remains open until one of the following occurs:
- The cursor is explicitly closed.
- An invalidate event occurs.
- If the deployment is a sharded cluster, a shard removal may cause an open change stream cursor to close, and the closed change stream cursor may not be fully resumable.
Note
The lifecycle of an unclosed cursor is language-dependent.
[1] | Starting in MongoDB 4.0, you can specify a startAtOperationTime
to open the cursor at a particular point in time. If the specified
starting point is in the past, it must be in the time range of the
oplog. |
Modify Change Stream Output¶
- Python
- Java (Sync)
- Node.js
- PHP
- Motor
- Other
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
The pipeline
list includes a single $match
stage that
filters any operations where the username
is alice
, or
operations where the operationType
is delete
.
Passing the pipeline
to the watch()
method directs the
change stream to return notifications after passing them through the
specified pipeline
.
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
The following example uses stream to process the change events.
Alternatively, you can also use iterator to process the change events:
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
You can control change stream output by providing an array of one or more of the following pipeline stages when configuring the change stream:
$addFields
$match
$project
$replaceRoot
$replaceWith
(Available starting in MongoDB 4.2)$redact
$set
(Available starting in MongoDB 4.2)$unset
(Available starting in MongoDB 4.2)
Tip
The _id field of the change stream
event document act as the resume token. Do not use the pipeline to modify or remove
the change stream event’s _id
field.
Starting in MongoDB 4.2, change streams will throw an exception if the change stream aggregation pipeline modifies an event’s _id field.
See Change Events for more information on the change stream response document format.
Lookup Full Document for Update Operations¶
By default, change streams only return the delta of fields during the update operation. However, you can configure the change stream to return the most current majority-committed version of the updated document.
- Python
- Java (Sync)
- Node.js
- PHP
- Motor
- Other
To return the most current majority-committed version of the updated
document, pass full_document='updateLookup'
to the
db.collection.watch()
method.
In the example below, all update operations notifications
include a full_document
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the updated
document, pass FullDocument.UPDATE_LOOKUP
to the
db.collection.watch.fullDocument()
method.
In the example below, all update operations notifications
include a FullDocument
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the updated
document, pass { fullDocument: 'updateLookup' }
to the
collection.watch()
method.
In the example below, all update operations notifications
include a fullDocument
field that represents the current
version of the document affected by the update operation.
The following example uses stream to process the change events.
Alternatively, you can also use iterator to process the change events:
To return the most current
majority-committed version of the updated document, pass
"fullDocument' => \MongoDB\Operation\ChangeStreamCommand::FULL_DOCUMENT_UPDATE_LOOKUP"
to the watch()
method.
In the example below, all update operations notifications
include a fullDocument
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the updated
document, pass full_document='updateLookup'
to the
db.collection.watch()
method.
In the example below, all update operations notifications
include a `full_document
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the updated
document, pass the "fullDocument"
option with the "updateLookup"
value to the
mongoc_collection_watch
method.
In the example below, all update operations notifications
include a fullDocument
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the updated
document, pass "FullDocument = ChangeStreamFullDocumentOption.UpdateLookup"
to the
collection.Watch()
method.
In the example below, all update operations notifications
include a FullDocument
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the updated
document, pass full_document: 'updateLookup'
to the
watch()
method.
In the example below, all update operations notifications
include a full_document
field that represents the current
version of the document affected by the update operation.
To return the most current majority-committed version of the
updated document, SetFullDocument(options.UpdateLookup)
change stream option.
To return the most current majority-committed version of
the updated document, pass options:
ChangeStreamOptions(fullDocument: .updateLookup)
to the
watch()
method.
To return the most current majority-committed version of
the updated document, pass options:
ChangeStreamOptions(fullDocument: .updateLookup)
to the
watch()
method.
Note
If there are one or more majority-committed operations that modified the updated document after the update operation but before the lookup, the full document returned may differ significantly from the document at the time of the update operation.
However, the deltas included in the change stream document always correctly describe the watched collection changes that applied to that change stream event.
See Change Events for more information on the change stream response document format.
Resume a Change Stream¶
Change streams are resumable by specifying a resume token to either resumeAfter or startAfter when opening the cursor.
resumeAfter
for Change Streams¶
You can resume a change stream after a specific event by passing a resume token
to resumeAfter
when opening the cursor. For the resume token, use the
_id
value of the change stream event document.
See Resume Tokens for more information on the resume token.
Important
- The oplog must have enough history to locate the operation associated with the token or the timestamp, if the timestamp is in the past.
- You cannot use
resumeAfter
to resume a change stream after an invalidate event (for example, a collection drop or rename) closes the stream. Starting in MongoDB 4.2, you can use startAfter to start a new change stream after an invalidate event.
- Python
- Java (Sync)
- Node.js
- PHP
- Motor
- Other
You can use the resume_after
modifier to resume
notifications after the operation specified in the resume
token. The resume_after
modifier takes a value that must
resolve to a resume token, e.g. resume_token
in the
example below.
You can use the resumeAfter()
method to resume
notifications after the operation specified in the resume
token. The resumeAfter()
method takes a value that must
resolve to a resume token, e.g. resumeToken
in the
example below.
You can use the resumeAfter
option to resume
notifications after the operation specified in the resume
token. The resumeAfter
option takes a value that must
resolve to a resume token, e.g. resumeToken
in the
example below.
You can use the resumeAfter
option to resume
notifications after the operation specified in the resume
token. The resumeAfter
option takes a value that must
resolve to a resume token, e.g. $resumeToken
in the
example below.
You can use the resume_after
modifier to resume
notifications after the operation specified in the resume
token. The resume_after
modifier takes a value that must
resolve to a resume token, e.g. resume_token
in the
example below.
In the example below, the resumeAfter
option is appended to the stream options
to recreate the stream after it has been destroyed. Passing the _id
to
the change stream attempts to resume notifications starting after the
operation specified.
In the example below, the resumeToken
is retrieved from the last change stream document
and passed to the Watch()
method as an option. Passing the resumeToken
to the Watch()
method directs
the change stream to attempt to resume notifications starting after the
operation specified in the resume token.
You can use the resume_after
modifier to resume
notifications after the operation specified in the resume
token. The resume_after
modifier takes a value that must
resolve to a resume token, e.g. resume_token
in the
example below.
You can use ChangeStreamOptions.SetResumeAfter
to specify the resume
token for the change stream. If the resumeAfter option is set,
the change stream resumes notifications after the operation
specified in the resume token. The SetResumeAfter
takes a
value that must resolve to a resume token, e.g.
resumeToken
in the example below.
You can use the resumeAfter
option to resume
notifications after the operation specified in the resume
token. The resumeAfter
option takes a value that must
resolve to a resume token, e.g. resumeToken
in the
example below.
You can use the resumeAfter
option to resume
notifications after the operation specified in the resume
token. The resumeAfter
option takes a value that must
resolve to a resume token, e.g. resumeToken
in the
example below.
startAfter
for Change Streams¶
New in version 4.2.
You can start a new change stream after a specific event by passing a resume
token to startAfter
when opening the cursor. Unlike
resumeAfter, startAfter
can
resume notifications after an invalidate event
by creating a new change stream. For the resume token, use the _id
value of
the change stream event document. See
Resume Tokens for more information on the resume token.
Important
- The oplog must have enough history to locate the operation associated with the token or the timestamp, if the timestamp is in the past.
Resume Tokens¶
The _id
value of a change stream event document acts as the resume token:
The resume token _data
type depends on the MongoDB versions and,
in some cases, the feature compatibility version (fcv) at the time
of the change stream’s opening/resumption (i.e. a change in fcv
value does not affect the resume tokens for already opened change
streams):
MongoDB Version | Feature Compatibility Version | Resume Token _data Type |
---|---|---|
MongoDB 4.2 and later | “4.2” or “4.0” | Hex-encoded string (v1 ) |
MongoDB 4.0.7 and later | “4.0” or “3.6” | Hex-encoded string (v1 ) |
MongoDB 4.0.6 and earlier | “4.0” | Hex-encoded string (v0 ) |
MongoDB 4.0.6 and earlier | “3.6” | BinData |
MongoDB 3.6 | “3.6” | BinData |
With hex-encoded string resume tokens, you can compare and sort the resume tokens.
Regardless of the fcv value, a 4.0 deployment can use either BinData resume tokens or hex string resume tokens to resume a change stream. As such, a 4.0 deployment can use a resume token from a change stream opened on a collection from a 3.6 deployment.
New resume token formats introduced in a MongoDB version cannot be consumed by earlier MongoDB versions.
Tip
Starting in MongoDB 4.2, change streams will throw an exception if the change stream aggregation pipeline modifies an event’s _id field.
Use Cases¶
Change streams can benefit architectures with reliant business systems, informing downstream systems once data changes are durable. For example, change streams can save time for developers when implementing Extract, Transform, and Load (ETL) services, cross-platform synchronization, collaboration functionality, and notification services.
Access Control¶
For deployments enforcing Authentication and authorization:
To open a change stream against specific collection, applications must have privileges that grant
changeStream
andfind
actions on the corresponding collection.To open a change stream on a single databases, applications must have privileges that grant
changeStream
andfind
actions on all non-system
collections in a database.To open a change stream on an entire deployment, applications must have privileges that grant
changeStream
andfind
actions on all non-system
collections for all databases in the deployment.
Event Notification¶
Change streams only notify on data changes that have persisted to a majority of data-bearing members in the replica set. This ensures that notifications are triggered only by majority-committed changes that are durable in failure scenarios.
For example, consider a 3-member replica set with a change stream cursor opened against the primary. If a client issues an insert operation, the change stream only notifies the application of the data change once that insert has persisted to a majority of data-bearing members.
If an operation is associated with a transaction, the change event document includes the
txnNumber
and the lsid
.
Collation¶
Starting in MongoDB 4.2, change streams use simple
binary
comparisons unless an explicit collation is provided. In earlier
versions, change streams opened on a single collection
(db.collection.watch()
) would inherit that collection’s
default collation.