codebyomar

Implement Elastic Search Using NestJS

January 30, 2019

Alt text I'm quite excited about this post, i have been thinking about the perfect topic to launch my blog with, after scribbling lots of topics i decided to write about Elasticsearch. The reason is because i recently implemented it in a side project, and it was quite interesting setting it up, and magical how fast the queries are. Let's go through it together.
If you are not familiar with elastic search don't get intimidated by the name. Elastic search is just a really, really fast RESTful search and analytics engine which can interface with various programming languages.
In this tutorial we are going to use NestJS as our backend to interact with our elastic search engine. NestJS is a progressive Node.js web framework built with TypeScript and uses ExpressJS under the hood.

Setting up elasticsearch cluster

Firstly we would create our elastic search server on Amazon Web Service (AWS). You can run a version locally or host it with a cloud service like DigitalOcean, AWS has a tailored service for hosting an elastic search engine which helps you get up and running in minutes. If you don't have an account on AWS, create one and read the following instructions to spin up your elastic search cluster.

Sign in to your console, under services > analytics, choose Elasticsearch service.
Choose create new domain.
Select a domain name, the latest version of Elasticsearch and click next.
Choose the number of instances you want, for this tutorial two instances are enough.
For Instance type select t2.small.elasticsearch (recommended for this tutorial).
Leave the other fields as the default values and click next.
For the set up access page, choose public access as the network configuration.
For the access policy, choose allow access to the domain from specific IP(s), and type your IP address, you can get your IP address by googling what is my IP.
Click next after inserting your IP.
Review your configuration and confirm, wait for a few minutes for the cluster to be active! If you run into any problems check AWS documentation.

Now lets create our backend that would communicate with our elastic search engine.

Prerequisites & Dependencies

Node.js
NPM or Yarn

If you have the above tools ready, go ahead and run the following commands to install the NestJS CLI and create a new nest project.

yarn global add @nestjs/cli # would install the nest command globally
nest new elasticsearch-tutorial # creates a new nest project in a new directory 'elasticsearch-tutorial'

Now navigate to the project directory to start the API:

cd elasticsearch-tutorial # navigates to the project directory
yarn start:dev # start the development server

Head to http://localhost:3000, you'll see a hello world page.

We are not going to be writing any tests in the tutorial so we are deleting the tests file in the test directory and in src/app.controller.spec.ts

We'll be needing a few dependencies, so let's go ahead and install them. There are two dependencies added by the command below, elasticsearch and @types/elasticsearch, elasticsearch is the client library that allow us to interface with our Elasticsearch cluster to perform API calls easily without needing to write much code, and the second dependency @types/elasticsearch contains type definitions for the library, it allows us to use the library in a very interactive and exploratory manner (also provides intellisense), which is one of the great benefits of TypeScript.

yarn add elasticsearch @types/elasticsearch # installs client library for elasticsearch and type definations for the library

Connecting to our cluster

After executing the above command successfully, we can now connect to our elasticsearch cluster and perform CRUD operations. Let's write the following code in our app.service.ts file.

import { Injectable, HttpException } from '@nestjs/common';
import * as elasticsearch from 'elasticsearch';

@Injectable()
export class AppService {
    private readonly esclient: elasticsearch.Client;

    constructor() {
        this.esclient = new elasticsearch.Client({
            host: '', // replace with your cluster endpoint
        });
        this.esclient.ping({ requestTimeout: 3000 })
        .catch(err => { 
            throw new HttpException({
                status: 'error',
                message: 'Unable to reach Elasticsearch cluster'
             }, 500); 
         });
    }
}

Let's digest the above code snippet. Firstly we imported Injectable and HttpException from @nestjs/common package, Injectable is a decorator that allows us to create providers in nestjs by simply annotating a class with the Injectable decorator. While the HttpException is a class that accepts two parameters, the custom response object and the appropriate HttpStatus. When a HttpException is caught by the built in handler, the custom object is transformed to JSON and sent to the user. The subsequent import is the elasticsearch library which we would need to interface with our elasticsearch cluster.
Next, we have our provider class AppService which is been exported. What we first did was to declare a variable esclient which would be our Elasticsearch client. Within the constructor block we then initialized our Elasticsearch client by passing in the host address. The next block of code pings our cluster to make sure we are in connection and the ping function returns a promise, which is been handled by our catch block. In the case of a failed connection a HttpException is thrown with the message 'Unable to reach Elasticsearch cluster'.

Creating Indices

An elasticsearch cluster consists of documents and indices. Each object stored within a cluster is known as a Document, which is a unit of information that can be indexed. Documents are synonymous to rows in relational database. One might ask how are these documents are organized or what is the equivalent of a relational database table in elasticsearch. Documents are stored within something called indices. An index is a collection of documents with similarly defined properties. The code snippets below creates an index and populates it with an array of documents and also searches the index:

 {
    // ... other codes here

    // loop through array and push the index and type of objects
    // create a 'pokemons' index and insert bulk documents
    async bulkInsert(abilities: any[]) {
        const bulk = [];
        abilities.forEach(ability => {
            bulk.push({ 
                index: {_index: 'pokemons', _type: 'abilities'} 
            });
            bulk.push(ability);
        });
        return await this.esclient.bulk({
            body: bulk, 
            index: 'pokemons', 
            type: 'abilities'
        })
        .then(res => ({status: 'success', data: res}))
        .catch(err => { throw new HttpException(err, 500); });
    }

    // searches the 'pokemons' index for matching documents
    async searchIndex(q: string) {
        const body = {
            size: 200,
            from: 0,
            query: {
                match: {
                    url: q,
                },
            },
        };
        return await this.esclient.search({index: 'pokemons', body, q})
            .then(res => res.hits.hits)
            .catch(err => { throw new HttpException(err, 500); });
    }
}

Although the above code snippets are quite descriptive, lets demystify a bit. The bulkInsert method accepts a single array parameter which is expected to contain abilities of pokemons, loops through it and pushes the name of the index and type of document the object is, this is required by elasticsearch in other to allow you to push documents of different types in bulk. While the searchIndex method as the name implies searches the pokemon index we created, it accepts a string to match against the url field of the various documents in our index.
Our AppService is now ready. One more thing we need to create are routes that will handle our requests and send response to clients. Open src/app.controller.ts and paste the following code in it:

import { Controller, Post, Body, Get, Query } from '@nestjs/common';
import { AppService } from './app.service';

@Controller('pokemon')
export class AppController {
    constructor(
        private readonly appService: AppService,
    ) {}

    @Post('create')
    async createIndexAndInsert(@Body() documents: any[]) {
        return await this.appService.bulkInsert(documents);
    }

    @Get('search')
    async searchPokemonAbilities(@Query('q') q: string) {
        const results = await this.appService.searchIndex(q);
        return results;
    }
}

Our controller exposes our service to clients. We created two endpoints in our controller by simply decrorating the methods with the appropriate HTTP method, createIndexAndInsert is accessible through the HTTP POST method and the searchPokemonAbilities is accessible through the HTTP GET method. Lets test our endpoints using postman.
We will need data to populate our index, so lets fetch some pokemon data from ThePokeAPI. Copy this link https://pokeapi.co/api/v2/ability?offset=0&limit=200 and post in your postman, it will return an array of two hundred pokemon abilities, copy the array, that would be enough to populate and test our elasticsearch index.
Next, open another postman tab:

Set the HTTP method as Post, enter this endpoint http://localhost:3000/pokemon/create to create our index and populate it.
Select the Body tab and paste the array you copied as a raw json
Send the request.

Your postman should something like this. alt text if it is successful you will receive a response with status as success. If you got a success message follow these steps to search the pokemon index;

Open a new postman tab an post this request url http://localhost:3000/pokemon/search
select the Param tab and type q as the key and "https" as the value.
send the request.

If you studied the pokemon abilities data we copied you would notice that each object has a url property. The above request would search for any the documents and return any whose url matches https.

you can check out the elasticsearch Query DSL to construct more complex queries.

Resources

I wanted to create a front end to consume the API we built but i guess that would be in another tutorial. You can check these links for more knowledge on NestJS and Elasticsearch;

Elasticsearch Docs: https://www.elastic.co/guide/index.html
NestJS Docs: https://docs.nestjs.com/

Personal blog by Umar Abdullahi

Get things done with mordern technologies.

← Go nuggets: Go code execution with time.Sleep