Leaistic: A Library and Microservice for Managing ElasticSearch Content

ElasticSearch is a great technology for a wide variety of use cases from autocompletion to log management, and is likely to be part of your stack for many complex projects.

In ElasticSearch, you put your data in indices. An index is a collection of documents, that shares the same mapping and settings.

We have found that Index management can be hard to manage over time:

1. ElasticSearch is not truly schemaless but it is smart enough to figure out the schema (aka datatype) most of the time. Thanks to that, you can play with ElasticSearch without configuring anything at first.
2. In a typical project, you will start with small documents, then add, delete and alter field content. You’ll also need to add some mappings to have control over the datatype of the fields. Then, you’ll probably need to update some field mapping at some point, only to discover you can’t, without re-indexing all of the index content. You’ll encounter the same issue, should you need to change your index static settings. Unfortunately if you did not prepare for it, this means production downtime.
3. ElasticSearch will figure out the datatype of a given field using the dynamic mapping rules it has. The first document you index containing a given field is its only context to figure out the right datatype.

During the life of a project, without being managed, an index will surely break.

A Workflow to Manage These Problems

In order to have the flexibility to change your index mappings or static settings whenever you want, you need to be able to create a new index with the same content and the new mappings and settings. Once reindexed, you need to use this new index instead of the original one. Ideally, this should happen with no downtime.

Keeping track of changing indices names at a given moment in your project can be hard to manage. ElasticSearch provides a solution to this with aliases: you can search and write to aliases like you do with indices. Aliases are just redirects to one or more indices, possibly with some filtering. They also allow you to redirect all of the queries from an index to a new one, with no downtime.

Here are simplified, nominal examples of what you should do to manipulate index-like alias foo:

• create a new index structure:
  
• update an existing index structure:
  
• delete an existing index structure:
  

One of the best ways to allow both machines and humans (for testing) to create indices that work in the long run, is to deploy index templates prior to any creations or updates with this workflow.

Introducing Leastic for ElasticSearch

Leaistic is both a library you can integrate, and a standalone REST microservice with a nice SwaggerUI.

Leaistic provides high-level primitives for creating, updating, and deleting indices, index templates, and aliases working together to help with the problems listed above. It does not cover all ElasticSearch possibilities like rollover index, multi-indices strategies, etc. If you start a new project with ElasticSearch, it will help you get your indices right.

Leaistic tries to be resilient against external changes in parallel with its operations, and has a per-alias locking mechanism to compensate for ElasticSearch’s lack of transactions for these operations. It is committed to providing helpful errors, and to rollback, when necessary, to the best possible state.

There is detailed usage information in the repo. Be sure to also read ‘Why use Leaistic’ and let us know what you think.

Read more about how we use JavaScript in our projects:

• How to build a real-time Node.js Performance Monitoring System
• Sematext Guest Post: PINO – The Fastest Node.js Logger for Production