Implement migrations in Mongo with Node

A migration in a database is defined as a series of steps or changes implemented in the database to modify the current schema or structure of the information. For example adding or removing fields or even changing specific data if certain conditions are met. Most ORM will have tools to create and execute migrations.

Migrations are important because it's a way we can alter the information in our database or add new fields to the existing schemas, for example lets say we have a schema called books, and we want to add a new a field called internal_id which is random string to identify the book, we would need to add this new field manually and create a script to change all entries in the database, with a migration we can achieve the same thing in a fraction of the time.

The problem is that in Node projects using Mongoose o MongoDB driver there is no way to implement migrations without adding a lot fo code. That's why we are going to use https://www.npmjs.com/package/migrate-mongo which is an npm package that will do the heavy lifting for us.

First we need to install the package

npm install migrate-mongo

Now run the following command to initialize migrate-mongo this will create a new file called migrate-mongo-config.js in the base of the project and a new folder called migrations where we will save the new migrations.

migrate-mongo init

This an example of the configuration file using env variables

//Set environment env variable
process.env["NODE_ENV"] = "development";

// Load env variables
require("dotenv-flow").config();

const config = {
    mongodb: {
        url: process.env.URL, // Mongo connection URL
        databaseName: process.env.MIGRATIONS_DATABASE, // the name of the database to execute the migrations
        options: {
            useNewUrlParser: true, // removes a deprecation warning when connecting
            useUnifiedTopology: true, // removes a deprecating warning when connecting
        },
    },

    // The migrations dir, can be and relative or absolute path. Only edit this when really necessary.
    migrationsDir: "migrations",

    // The mongodb collection where the applied changes are stored. Only edit this when really necessary.
    changelogCollectionName: "changelog",

    // The file extension to create migrations and search for in migration dir
    migrationFileExtension: ".js",

    // Enable the algorithm to create a checksum of the file contents and use that in the comparison to determin
    // if the file should be run.  Requires that scripts are coded to be run multiple times.
    useFileHash: false,
};

// Return the config as a promise
module.exports = config;

Now we can run the following command to create a migration, this will add a new file inside the migrations folder

npx migrate-mongo create migration-name

In this new file we can add code in the up function to edit the database or add new data, we also have access to the down function which we will use to revert all the changes done in the up function.

const ObjectID = require("mongodb").ObjectID;

// up function will execute the logic for the migration
// down function reverts up function changes
module.exports = {
async up(db, client) {
},

async down(db, client) {},
};

Let's change the migration to add a new field to the Books collection, we will also delete an old field to demonstrate how to do this

const ObjectID = require("mongodb").ObjectID;

module.exports = {
    async up(db, client) {
        const database = client.db("Library");
        // Get Library database, we could get more databases if we need to
        const allBooks = await database
          .collection("Books")
          .find({})
          .toArray();

        // Move through all the Books in the database
        for (const singleBook of allBooks) {
            await database.collection("Books").updateOne(
              { _id: new ObjectID(singleBook._id) },
              {
                  $set: {
                      internal_id: Math.random().toString(36).slice(2, 12), // Set internal id to ramdom string
                  },
                  $unset: {
                      old_field: "", // Remove old field
                  },
              }
            );
        }
    },

    async down(db, client) {
        // Get Library database, we could get more databases if we need to
        const allBooks = await database
            .collection("Books")
            .find({})
            .toArray();

        // Move through all the Books in the database
        for (const singleBook of allBooks) {
            await database.collection("Books").updateOne(
                { _id: new ObjectID(singleBook._id) },
                {
                    $unset: {
                        internal_id: "", // Remove internal id added in the up function
                    },
                }
            );
        }
    },
};

Now run the following command to execute the pending migrations

npx migrate-mongo up

If you want to revert the last executed migrations, run the following command

npx migrate-mongo down

If you want to see the migrations that were already executed you can see them in a new collection called "changelog" created by migrate-mongo