Accessing the MongoDB oplog

Overview

All for-pay plans with mLab have access to the “local” database which contains the oplog (operations log), a special capped collection that keeps a rolling record of all insert, update and delete operations.

The oplog provides valuable information that can be used in a variety of applications. For example, Meteor-built websites rely on the oplog to get real-time updates on data changes. Search and analytic engines such as Elasticsearch (via the mongo-connector) can also use the oplog to sync your data and help you gain insight from all of your application’s data-related activities. These are just two of the many applications and libraries out there that are designed to take advantage of an oplog’s contents.

Since the oplog is critical to replication and the health of a cluster, we recommend against querying the oplog on the primary node of a cluster. Oplog queries are resource intensive because they are not able to utilize indexes and require full collection scans for each operation. Regularly querying the oplog can severely impact performance and even cause downtime.

Tailing the oplog by using tailable cursors is generally the recommended approach for syncing data from the oplog.

Accessing the oplog

Not available for Sandbox databases

Creating a database user to access the oplog

For any application to read or tail the oplog, you will need to create a database user as per the following instructions:

  1. Log in to the mLab management portal.
  2. From your account’s Home page, navigate to the deployment whose oplog you want to access.
  3. Click the “admin” database under the “System Databases” section. img-admindb
  4. Click the “Users” tab.
  5. Click the “Add oplog user” button to create a new user.
    img-addoploguser

Connecting to the oplog

To connect to the oplog, you’ll need to make sure that your connection string is pointing to the “local” database where the oplog resides but authing against the “admin” database because the oplog reader database user was created in that database.

For example, your connection string may look like this:

mongodb://<dbuser>:<dbpassword>@ds012345-a0.mlab.com:56789,ds012345-a1.mlab.com:56790/local?replicaSet=rs-ds012345&authSource=admin

Frequently Asked Questions

Q. Does the “local” database holding the oplog count towards my quota and/or bill?

Shared plan deployments are currently configured with a 2 GB oplog, and this oplog is included for free. In other words, the local database does not contribute to the quota or bill for Shared plan deployments.

Dedicated plans are initially configured using MongoDB’s default oplog size but the size can then be customized. The oplog on Dedicated plan deployments does count toward the pre-configured amount of storage that comes with each plan. For example, if your Dedicated plan includes 60 GB of storage and the oplog has been configured to be 10 GB, there will only be 50 GB left for use.

Q. How do I create a database user that can read both the oplog and my database?

Some client libraries such as mongo-connector need access not only to the oplog, but also to the database itself. If you are in this situation, please email support@mlab.com with the following two pieces of information:

  1. Your deployment identifier (e.g., dsXXXXXX)
  2. The name of the oplog database user you’ve already created

Dedicated plan deployments

Our Ops team will run the following to grant more privileges to your existing oplog reader user:

db.grantRolesToUser(OPLOG_READER_USER, ["readAnyDatabase"])

Shared plan deployments

Our Ops team will run the following to grant more privileges to your existing oplog reader user:

db.grantRolesToUser(OPLOG_READER_USER, [{"role": "read", "db": MY_DB}] )

This will soon be added as a self-service function in our management portal but in the meantime, please email support@mlab.com as instructed above.