Migrating Data Into mLab
- Migrating from an existing MongoDB deployment
- Importing a JSON, CSV or TSV file
- Pre-filled import/export command-line strings
mLab is now part of MongoDB, Inc., the company behind the database that powers your application(s). For background please read our announcement on October 9, 2018.
As part of merging the two organizations we are sunsetting mLab’s service and working with users to migrate to MongoDB Atlas. As such it probably makes more sense to migrate to MongoDB Atlas which offers a similar service.
Migrating from an existing MongoDB deployment
There are several different ways to migrate data into mLab. The method you choose depends on your particular data source and your uptime requirements.
If you have any questions about which process makes the most sense for your application, don’t hesitate to contact support@mlab.com for help.
Use mLab’s migration tool
This feature is currently in closed beta for eligible Dedicated plans only.
If you are currently running a MongoDB instance that you would like to migrate over to a mLab-hosted deployment, you have the option to migrate with almost no downtime using mLab’s migration tool.
Pre-requisites for Source Deployment
- Must be running version 2.6.x, 3.0.x, 3.2.x, or 3.4.x
- Some restrictions apply if running version 2.6.x that is prior to 2.6.10
- Must be running as a Replica Set
- Must be able to supply mLab with an admin database user of the Replica Set that has least the privileges of the backup role
- Must be able to allow inbound network access from a mLab-hosted deployment
To migrate data from another MongoDB deployment, follow these steps:
- Log in to the mLab management portal.
- Create your new target deployment on mLab if you haven’t already done so.
- If you’re migrating into one of our Dedicated plans, we recommend creating a larger plan than you think you’ll end up on to help with the initial bulk load. We pro-rate charges by the day and downgrades are seamless on our Dedicated Cluster plans via our rolling node replacement process.
- Make sure the MongoDB version matches the version of the deployment that you are migrating from.
- When you are ready to start the migration, navigate to the target deployment from your account’s Home page.
- Click the “Tools” tab.
- Click the “Migrate from another deployment” button.
- If migrating from a deployment that is not also associated with the current mLab account, fill in the information in the option labeled “Migrate from a remote deployment”. Otherwise, select the option “Migrate from another deployment in the same account” and choose the desired deployment.
- Click the “Live migrate” button - a confirmation window will appear for you to confirm that you want to start the migration.
- Once the migration has started, you can watch its progress.
- The migration tool will take an initial copy of your deployment and restore it into your mLab-hosted deployment.
- After that, it will keep up-to-date with your deployment by tailing the source deployment’s oplog.
- When the progress window indicates that the migration has entered the keep-up phase and that the deployments are in sync, you can stop the migration by clicking the “Stop migration” button.
- After you have stopped the migration, confirm that you can connect and authenticate to your target deployment and check the data that has been migrated.
Do not execute writes against the target deployment until the migration is complete or else there could be data integrity problems.
Using replica set replication
Available for Dedicated plans only.
If you are currently running a replica set that you would like to migrate over to one of mLab’s Dedicated plans, you have the option to minimize downtime by using MongoDB’s internal replication mechanisms.
Pre-requisites for the source deployment
- Must be running version 2.6.x+, 3.0.x, 3.2.x, or 3.4.x
- Must be running as a replica set
- If running in “auth” mode, must be able to supply mLab with the replica set’s key file (the value for the
--keyFile
option) - Must be able to allow network access from and to a mLab-hosted deployment
Using mongodump / mongorestore
MongoDB’s mongodump and mongorestore utilities are command-line tools that provide fine-grained data export and import capabilities. To download these utilities visit the MongoDB download page.
When you are ready to copy data from one MongoDB deployment to another, follow these instructions:
- Log in to the mLab management portal.
- If you have not already done so, create a new mLab deployment as your target.
- Be sure to create database user(s) on the new deployment.
- To prevent data inconsistency, stop any processes running against your source deployment.
- Take a backup your source deployment by running the
mongodump
command in a terminal window.- See below for a single-database example.
- Restore the backup into your target deployment by running the
mongorestore
command.- See below for single-database example.
- Also, mLab provides pre-filled command-line strings for import/export commands to help.
- Update your application with the target’s connection URI and start writing to your new deployment.
Example
To mongodump
a single database from your source deployment:
% mongodump -h source_host:source_port -d dbname -u source_dbuser -p source_dbpassword -o dump_dir
To mongorestore
this backup to your target deployment:
% mongorestore -h target_host:target_port -d dbname -u target_dbuser -p target_dbpassword dump_dir/*
To minimize the possibility of error, the versions of your target database, source database, and mongodump/mongorestore utility should match. For example: use mongodump 3.0 to restore a backup taken from MongoDB 3.0 into a MongoDB 3.0 database.
Note that the version of your mongodump/mongorestore utility must match the version of MongoDB you have deployed. If you have multiple MongoDB versions installed, be sure to use the correct one.
Using copydb
Available for Dedicated plans only
MongoDB’s copydb
command allows you to copy a database directly from a remote, source instance to the current, destination instance.
This method is faster than mongodump/mongorestore, but you must have a Dedicated plan with mLab in order to run this command, since it requires full administrative privileges on the destination instance.
- Log in to the mLab management portal.
- If you have not already done so, create a new mLab deployment to host the destination database.
- The destination database itself does not need to be created; the command will implicitly create it if it does not exist.
- If you have not already done so, create an admin database user for the destination deployment.
- To prevent data inconsistency, stop any processes running against your source database.
- In a terminal window, connect to the destination instance using the admin database user’s credentials.
- Run the
db.copyDatabase()
helper method.- The expected order of arguments is somewhat confusing so take caution as you construct the command.
- The duration of the copy process can vary widely, depending on the amount of data, number of indexes, network latency, etc..
- When the copying has completed and you have updated your application with the new connection URI, you can start writing to your new database.
If you have SSL enabled on your deployment, note that copydb
will not work unless the remote source also supports SSL.
Importing a JSON, CSV or TSV file
If you need to import or export data in a flat file (non-binary) format such as JSON, CSV, or TSV, the mongoimport
and mongoexport
commands are the appropriate MongoDB command-line tools to use. To download these utilities, visit the MongoDB download page.
Once MongoDB is installed successfully, follow the instructions below to import a JSON, CSV, or TSV file into your mLab-hosted database:
- Log in to the mLab management portal.
- If you have not already done so, create a new mLab deployment as your target.
- Be sure to create database user(s) on the new deployment.
- Use the
mongoimport
command to import your text file(s) into your new deployment.- Take advantage of mLab’s pre-filled command-line strings for import/export commands and choose the JSON- or CSV-specific line to get you started with constructing the command.
Example
To mongoimport
a JSON file to a collection in your target deployment:
% mongoimport -h ds012345.mlab.com:56789 -d dbname -c collectionname -u dbuser -p dbpassword --file filename.json
To mongoimport
a CSV file to a collection in your target deployment:
% mongoimport -h ds012345.mlab.com:56789 -d dbname -c collectionname -u dbuser -p dbpassword --file filename.csv --type csv --headerline
- The first row of the CSV input file must specify the field names; each subsequent line in the file must represent a single document.
- The
--headerline
option in the command is what tellsmongoimport
to interpret the first line of the CSV file as the field names and not as a document. - For a TSV file, all of the above also applies, the only difference is that
--type tsv
should replace--type csv
in the command.
Pre-filled import/export command-line strings
MongoDB commands can be tricky to write and typos aren’t any fun so we’ve tried to make it easier for you by constructing pre-filled command-line strings that you can copy and paste directly into your terminal. To obtain these strings, follow these instructions:
- Log in to the mLab management portal.
- From your account’s Home page, navigate to the deployment with the database that you are going to import or export.
- If necessary, navigate to the desired database after navigating to the deployment.
- Click the “Tools” tab.
- Click the “import/export” link and find the command you want to use.
- Replace all placeholders (i.e., the values enclosed in angled brackets).
- Make sure the database user is NOT a read-only user.
- The <input db directory> for mongorestore is not the same as the <output directory> for mongodump. The <input db directory> should be one level below the <output directory>. For example, if the mongodump <output directory> was “dump_dir” for the database “foo”, the <input db directory> for mongorestore would be “dump_dir/foo”.
- In your terminal window, run the adjusted command.
The strings provided in the tool are basic and do not include many of the extra options associated with the MongoDB commands. You may need to adjust the strings further, depending on which options you desire to use.