Setting Cron Jobs in a project using PM2

PM2 is a process manager for node js applications that allows you to run multiple instances of a project. Instead of having just one application listening for incoming requests you will have multiple. This allows you to have 0 downtime because if one instance fails or encounters and error the other ones can handle the remaining traffic. PM2 also includes a load balancer, so you don't need to configure this yourself.

When trying to set up cron jobs using node-cron or a similar package in a project using PM2 we will see the jobs are running multiple times. The problem is that the jobs are being initialized one time per each pm2 instance. So if we have 6 pm2 instances then our cron jobs will run 6 times. Let's say we are sending an email to our users in one of this cron jobs then we will spam the user email account.

Thanks to this link I was able to find this solution. The idea is simple in our ecosystem.config.js file which is the file that tells PM2 the applications to initialize and the number of instances for each application. We need to divide our app in two different applications the first one is going to be the primary application and the one in charge of running the cron jobs that's why we set the instances property to 1, so PM2 only creates one instance of this app. And the second one will be the replica in this one we set to -1 the instances property to tell PM2 to initialize the maximum amount available.

module.exports = {
    apps: [
        {
            script: "./build/index.js",
            instances: "1",
            exec_mode: "cluster",
            name: "primary",
            env: {
                NODE_ENV: "development",
            },
            env_integration: {
                NODE_ENV: "integration",
            },
            env_production: {
                NODE_ENV: "production",
            },
        },
        {
            script: "./build/index.js",
            instances: "-1",
            exec_mode: "cluster",
            name: "replica",
            env: {
                NODE_ENV: "development",
            },
            env_integration: {
                NODE_ENV: "integration",
            },
            env_production: {
                NODE_ENV: "production",
            },
        },
    ],
};

Then we can initialize our cron job only if the env variable name is primary

const cron = require('node-cron');

// pm2 instance name
const processName = process.env.name || 'primary';

// Only schedule cron job if it´s the primary pm2 instance
if(processName === 'primary'){
    // schedule cron job
    cron.schedule('* 15 * * * *', () => {
        console.log('node-cron cron job');
    });
}

BullMQ is a node package that allows us to use a queue system built on top on Redis. It contains a lot of functionalities, but the ones we will use are repeatable jobs and rate limit. Rate limit will tell the queue, the maximum amount of times a specific job can be run at the same time, so if we set this option to 1 it doesn't matter if we have multiple pm2 instances we will execute the job only one time.

const IORedis = require('ioredis');
const { Job, Queue, Worker } = require("bullmq");

  // Initialize redis connection
const connection = new IORedis('redis://127.0.01:6379');

// Create queue to run cron jobs using redis connection
const queue = new Queue("cron", {
    connection
});

/**
 * Add job to queue with the data and cron syntax for the repeat pattern and specify limit to 1, this avoids
 * running this cron job more than one time
 */
await queue.add(
    'job1',
    { cronNumber: 1 },
    {
        repeat: {
            pattern: '* 15 * * * *',
            limit: 1,
        },
    },
);

// Create worker to specify the functionality the job1 needs to execute
new Worker(
    "cron",
    async (job) => {
        console.log(`BullMQ cron job`)
    },
    {connection, autorun: true}
);

Agenda is another node package that allows us to run specific jobs or functions in a given time. The difference with BullMQ is that agenda is more focused in scheduling jobs than queueing them, also agenda uses MongoDB instead of Redis to store data. When using Agenda we are interested in the concurrency option. Concurrency specifies the maximum amount of that job that can be running at once, by setting the option to 1, we ensure we only run our job once, even though we have multiple pm2 instances.

const Agenda = require("agenda");

const mongoConnectionString = "mongodb://127.0.0.1/agenda";

// Initialize agenda with mongo connection string
const agenda = new Agenda({ db: { address: mongoConnectionString } });

/**
 * Specify the functionality the job1 needs to execute, and setting concurrency to 1, this avoids running this cron job
 * more than one time
 */
agenda.define("job1", { concurrency: 1 }, (job) => {
    console.log('Agenda cron job')
})

// Start agenda
await agenda.start();

// Schedule job1
await agenda.every("* 15 * * * *", "job1");

In this repository https://github.com/obravocedillo/PM2CronJobs you can find the code implementing all the solutions mentioned. Just download the code, install all dependencies using npm install and use the command npm run initialize-pm2 you will see we initialize the project using pm2 in cluster mode. Every 5 minutes the cron jobs will run, logging to the console, we will see only one message per job proving we are running the jobs only once even though we have multiple pm2 instances.