Skip to content

Using Atlas search index to query data on aggregation pipelines

What's Atlas search index?

Atlas search index is a special type of index that allows us to run full-text search queries with MongoDB. For example if we want to add an autocomplete feature or see if a field in the database contains a specific string or sub string, we wil normally do it using regular expressions, which have a really poor performance. With this index we could do the same thing much faster and more efficient. In the background this index uses Apache Lucene as a full-text search engine.

Creating a new Atlas search index

To create a new index first all, our database must be hosted in Atlas, which is the official cloud solution for MongoDB, secondly our database needs to be in a dedicated server plan, we cannot create a new index using shared or serverless plans.

To create a new search index we have two options, we can click the text bellow "ATLAS SEARCH" in the right bottom corner of the main database page. This will redirect us to the search tab of the database options or we can directly go this tab ourselves. Mongo database initial dashboard Mongo search tab option

Now click "CREATE INDEX", this will open the index configuration wizard, the first step is selecting the configuration type. We will use the visual editor for this tutorial. The second step is selecting the name, database and collection we will use. In the next step we will see a summary of the configuration of the new index, click "Refine your index" button below the summary. This will allow us to do further configurations to the index.

Refine index

On the refine window, turn off the "Dynamic Mapping" option and in the "Field Mapping" section add all the fields you want to use to search or filter the data. I will explain the data types in the next section. Once you add all the information needed click "Save Changes" and the index will be created.

Refine index

Using Atlas search indexes in queries

To use search indexes we need to use the $search aggregation pipeline stage instead of a normal find query. Now this pipeline has special operators depending on the fields we are using to search or filter the information, you can find a list with all the available operators in the following link https://www.mongodb.com/docs/atlas/atlas-search/operators-and-collectors/ I will explain how to use some of these in the next examples.

Important remember to add all the fields you want to use in the queries to the "Field Mappings" section in Atlas, every field type will have specific operators that can be used. For example equals only works for booleans and objectId.

MongoDB Node Driver Examples

/*
 * Search in a string field using the autocomplete operator
 */
let usersSingle = await db.collection('Users').aggregate([
    {
        $search: {
            index: "articles_description",
            autocomplete: {
                query: searchInput,
                path: "description",
            },
        },
    },
]).toArray();

/*
 * Search in two string fields combining both autocomplete operators,
 * we use the compound operator to merge multiple operator into a single
 * query. Inside of compound we have multiple options depending if we want
 * to affect how the query scores the results
 */
let usersCombining = await db.collection('Users').aggregate([
    {
        $search: {
            index: "user_index",
            compound: {
                /*
                * Documents that contain a match for a should clause have higher
                * scores than documents that don't contain a should clause.
                */
                should: [
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "description",
                        },
                    },
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "name",
                        },
                    },
                ],
            },
        },
    },
]).toArray();


/*
 * Search in two string fields combining both autocomplete operators,
 * user must be active and their age must be inside of the range,
 * instead of using $match as in normal aggregation we use filter
 */
let usersCombiningFilters = await db.collection('Users').aggregate([
    {
        $search: {
            index: "user_index",
            compound: {
                /*
                * Documents that contain a match for a should clause have higher
                * scores than documents that don't contain a should clause.
                */
                should: [
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "description",
                        },
                    },
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "name",
                        },
                    },
                ],
                filter: [
                    {
                        equals: {
                            // Boolean value to use in the filter
                            value: activeUser,
                            path: "active",
                        },
                        // We can use one or multiple gte, gt, lte, lt options
                        range: {
                            // Number value to use in the filter
                            gte: minimumAge,
                            // Number value to use in the filter
                            lte: maximumAge,
                            path: "age",
                        }
                    },
                ],
            },
        },
    },
]).toArray();

/*
 * Search in two string fields combining both autocomplete operators,
 * user must be active and their age must be inside of the range,
 * paginate results
 */
let usersComnbiningFiltersPaginated = await db.collection('Users').aggregate([
    {
        $search: {
            index: "user_index",
            compound: {
                /*
                 * Documents that contain a match for a should clause have higher
                 * scores than documents that don't contain a should clause.
                 */
                should: [
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "description",
                        },
                    },
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "name",
                        },
                    },
                ],
                filter: [
                    {
                        equals: {
                            // Boolean value to use in the filter
                            value: activeUser,
                            path: "active",
                        },
                        // We can use one or multiple gte, gt, lte, lt options
                        range: {
                            // Number value to use in the filter
                            gte: minimumAge,
                            // Number value to use in the filter
                            lte: maximumAge,
                            path: "age",
                        }
                    },
                ],
            },
        },
    },
    // Skip certain amount of results
    { $skip: skipAmount },
    // Return certain amount of results
    { $limit: limitAmount },
]).toArray();

Mongoose Examples

/**
 * Users is a mongoose model containing the schema for the User
 * collection
 */


/*
 * Search in a string field using the autocomplete operator
 */
let usersSingle = await Users.aggregate([
    {
        $search: {
            index: "articles_description",
            autocomplete: {
                query: searchInput,
                path: "description",
            },
        },
    },
]).exec();

/*
 * Search in two string fields combining both autocomplete operators,
 * we use the compound operator to merge multiple operator into a single
 * query. Inside of compound we have multiple options depending if we want
 * to affect how the query scores the results
 */
let usersCombining = await db.collection('Users').aggregate([
    {
        $search: {
            index: "user_index",
            compound: {
                /*
                * Documents that contain a match for a should clause have higher
                * scores than documents that don't contain a should clause.
                */
                should: [
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "description",
                        },
                    },
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "name",
                        },
                    },
                ],
            },
        },
    },
]).exec();


/*
 * Search in two string fields combining both autocomplete operators,
 * user must be active and their age must be inside of the range,
 * instead of using $match as in normal aggregation we use filter
 */
let usersCombiningFilters = await Users.aggregate([
    {
        $search: {
            index: "user_index",
            compound: {
                /*
                * Documents that contain a match for a should clause have higher
                * scores than documents that don't contain a should clause.
                */
                should: [
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "description",
                        },
                    },
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "name",
                        },
                    },
                ],
                filter: [
                    {
                        equals: {
                            // Boolean value to use in the filter
                            value: activeUser,
                            path: "active",
                        },
                        // We can use one or multiple gte, gt, lte, lt options
                        range: {
                            // Number value to use in the filter
                            gte: minimumAge,
                            // Number value to use in the filter
                            lte: maximumAge,
                            path: "age",
                        }
                    },
                ],
            },
        },
    },
]).exec();

/*
 * Search in two string fields combining both autocomplete operators,
 * user must be active and their age must be inside of the range,
 * paginate results
 */
let usersComnbiningFiltersPaginated = await Users.aggregate([
    {
        $search: {
            index: "user_index",
            compound: {
                /*
                * Documents that contain a match for a should clause have higher
                * scores than documents that don't contain a should clause.
                */
                should: [
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "description",
                        },
                    },
                    {
                        autocomplete: {
                            // Variable we want to use for the search
                            query: searchInput,
                            path: "name",
                        },
                    },
                ],
                filter: [
                    {
                        equals: {
                            // Boolean value to use in the filter
                            value: activeUser,
                            path: "active",
                        },
                        // We can use one or multiple gte, gt, lte, lt options
                        range: {
                            // Number value to use in the filter
                            gte: minimumAge,
                            // Number value to use in the filter
                            lte: maximumAge,
                            path: "age",
                        }
                    },
                ],
            },
        },
    },
    // Skip certain amount of results
    { $skip: skipAmount },
    // Return certain amount of results
    { $limit: limitAmount },
]).exec();