Elastic 2 Atlas Search: A Journey

There are scenarios when customers would require a unified database for supporting both transactional and search. With the advent of Atlas Search this makes that possible. Below are the things to consider

1. Schema Design
2. Data Migration
3. Elastic Search Query Patterns
4. Application Migration
5. Provision to convert Elastic Query to Atlas Search Query

In this blog we will look at the design of application migration approach and a tool for converting Elastic to Atlas Search Queries

What is Atlas Search ?

Atlas Search is an embedded full-text search in MongoDB Atlas that gives you a seamless, scalable experience for building search on top of your data with an integrated, fully managed search engine that automatically syncs to your database. Built on Apache Lucene, Atlas Search eliminates the need to run a separate search system alongside your database.

Below is a typical Atlas Search Architecture. The Atlas Search mongot Java web process uses Apache Lucene and runs alongside mongod on each node in the Atlas cluster. The mongot process

1. Creates Atlas Search Indexes
2. Monitors Change Stream of the collection for which Atlas search indexes are defined.
3. Process Atlas Search queries and returns matching documents

Atlas Search Architecture : Credits MongoDB

What is Elastic Search ?

Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene and developed by Elastic.It is an external search engine which needs to be introduced into your application infrastructure needing to mandate synchronisation mechanism that replicates data from the database to the search engine.

Below is a typical Elastic Search Architecture.

• Relations database is your read/write source of truth whilst Elasticsearch is your read-only search layer.
• You have data that is constantly changing and have Sync System in place to replicate the records as documents into Elasticsearch for text-based or auto-complete queries.

Elasticsearch Architecture

General Application Architecture

The typical application architecture would involve a web application to frame the search requests, later the backend API converts the search requests to native Elastic Query and runs the query using ElasticClient.

Elastic Flow

1. The clients would send the search requests to the web APIs.
2. The Service layer interprets the requests to elastic search query
3. The Data Access layer would leverage elastic client to execute the search against the elastic cluster

Atlas Search Flow

1. The clients would send the search requests to the web APIs.
2. The Service layer would be rewritten to convert the requests to Atlas search query.
3. The Data access layer would be rewritten to leverage Mongo Client and execute the search request against MongoDB.

Elastic2Atlas Transpiler

Migration developers would need to quickly analyse the various elastic search query patterns and check the feasibility of converting Elastic to Atlas search queries.

Elastic2Atlas plugin is an ANTLR based transpiler for converting Elastic to Atlas Queries and will help developers coming from Elastic ecosystem during initial days until they get comfortable with Atlas Search.

Design

1. Feed the elastic query as an input to the library.
2. Define grammar file to traverse the elastic query and generate parse tree.
3. Programmatically generates a skeleton visitor classes from ANTLR.
4. ANTLR-generated visitor class, these methods do not do anything except recurse on the child nodes.
5. In order for our visitor to be able to transpile code, we need to subclass the generated visitor class and override each visit method to define what to do with each type of node
6. Generate Atlas Search Query to be executed as a part of aggregation pipeline.