Wondering how to get Elasticsearch and Firebase’s Firestore to play nice?
4 min read

Wondering how to get Elasticsearch and Firebase’s Firestore to play nice?


If you just came here for a solution and not some discourse, jump to the repo: https://github.com/acupofjose/elasticstore, though the exposition should clear the thought process up!

Want to see it in action? Check out: https://elasticstore.netlify.app


Everyone else, you came to the right spot! We had the same question on one of our passion projects Catholify. Specifically, in finding places that Firestore doesn’t offer functionality that a more seasoned product would. But hey, they don’t call it the bleeding edge for nothing.

Limitations we found:

Enter Elasticsearch.

With Elasticsearch, provided you can hydrate data back-and-forth from Firebase, we can fix those problems.

Enough intro. Let’s see how this works.


For the sake of brevity, we’ll do a sandbox setup. Assumptions on the following are a Linux based system with Docker, Git and Node.js installed.

Roadmap

  1. Set up Elasticsearch
  2. Set up Firebase
  3. Communicate between them using Elasticstore.
  4. Run searches leveraging Firestore
  5. Caveats

Setting up Elasticsearch

Fire up that terminal.# Get Elasticsearch running.

$ git clone https://github.com/deviantony/docker-elk
$ cd docker-elk
$ docker-compose up -d

Poof. Just like that we’re off to the races. Yay docker!

That command will make Elasticsearch (http://localhost:9200), Kibana (http://localhost:5601), and Logstash (we don’t care about right now) available locally.


Setting up Firebase

So that gives us Elasticsearch, now we need the Firebase side.

  1. Under Database in the Firebase console, start a new (if you don’t already have one) firestore database. Put some data in there that you want to use.
  2. We need some service account credentials: Firebase Console Project Settings Service Accounts Node.js (selected) Generate New Private Key.

For the sake of this example we will be using the following data://Root Collection: Users

{  
    "AWGSLQhnug3dSHq0nECo": {    
        "firstName": "John",    
        "lastName": "Doe",    
        "location": {        // <-- This is how a GEOPOINT is stored      
            "_latitude": 40.7128,
            "_longitude": -74.0060    
        } 
    },  
    "BASQWERhnug3dS126d1": {    
        "firstName": "John",    
        "lastName": "Doe",    
        "location": {      
            "_latitude": 37.7749,
            "_longitude": -122.4194    
        }  
    }
}

* Note: Location is a geopoint in Firebase’s eyes. The ids don’t matter, that’s just what Firestore generated.

Great. Now that we’ve got something to query against.

Keep your Firebase console open!


Communicating between them using Elasticstore

Now for the key to all of it.$ git clone https://github.com/acupofjose/elasticstore
$ cd elasticstore
$ yarn install$ cp .env.sample .env

Now we need to get our environment set up.

  1. Copy your service-account.json credentials into the elasticstore folder.
  2. Open up .env and plug in your variables it should end up something like this:# Firebase
FB_URL={{YOUR FIRESTORE URL}}
FB_ES_COLLECTION="search"
FB_REQ="request"
FB_RES="response"
FB_SERVICE_PATH="service-account.json"
ES_HOST=localhost
ES_PORT=9200

Wonderful. Now we just need to tell Firestore what data to send to Elasticsearch.

In src/references.ts you’ll find the place to do that. In our case, we want to send the whole user over to Elasticsearch for indexing, but would like to provide a mapping from Firebase’s Geopoint data-type to Elasticstore’s geo_point data-type.

So we’ll modify src/references.ts to suit:

[  
    {    
    	collection: "users",    
        index: "users",    
        type: "users",    
        mappings: {      
        	location: {
            	type: "geo_point" // Elasticsearch's 'geo_point' needs to be specified     
            }    
        },
        // Transform data as we receive it from firestore    
        transform: (data, parent) => ({
        	...data,
        location:`${doc.location._latitude},${doc.location._longitude}`
        })
    }
]

Now just to start the script! npm run start

And that’s it! Firestore will now sync ALL user documents, and their changes, while coercing them into something that Elasticsearch understands.


Running Searches leveraging Firestore (Client Side)

While I was looking over the Flashlight repo by the Firebase Team, I also decided to migrate over their convention of performing a search.

Elasticstore will listen to the search/ root collection for a new document containing a request object key and a null response object key. Upon finding a request that is ‘unfulfilled’ (a null response).

Requests should be formed as new documents in the search collection.

Assuming you’re using the node.js Firebase SDK, making an Elasticsearch request through Firebase would look something like this:

const result = await firebase.firestore().collection('search').add({  
	request: {    
        index: 'users',    
        type: 'users',    
        q: 'John' // Shorthand query syntax  
    },  
    response: null 
})

result.ref.onSnapshot(doc => {  
	if (doc.response !== null) {  
    	/* Do things */  
    }
})

Or with the normally expected Elasticsearch syntax body:

const result = await firebase.firestore().collection('search').add({
	request: {
        index: 'users',
        type: 'users',
        body: {
            query: {
                match: {      
                    "_all": "John"         
                }
            }
    	},
    },
    response: null
})

result.ref.onSnapshot(doc => {  
    if (doc.response !== null) {    
    	// Do things  
    }
})

Elasticstore will search for fulfilled request and delete them on a timeout.

But there we have it! An Elasticsearch stack, connected to Firestore with updates, and agnostically searchable by leveraging Firestore’s listeners.


Caveats

As with any solution, this comes with caveats.

  1. Only data that you want to be publicly query-able should be stored in Elasticsearch. I don’t know of a good way to restrict searches made against Elasticsearch through Firestore without it just being security-through-obscurity.
  2. You should check out the SearchGuard branch of docker-elk and adjust your code accordingly to add some security to the Elasticsearch stack.
  3. Once documents have been added to Elasticsearch’s indexes, it would make the most sense to only insert/update documents that are recent. You can do this by maintaining a field on the document i.e. updatedAt, and adjusting your references.ts to something like:
    builder: (q) => q.where('updatedAt', '>=', INSERT_TIMEFRAME_HERE) Otherwise you will end up with a large number of documents always being listened to, which for large databases, may cause some issues.
  4. This script handles BOTH listening and search requests, separating them out into different handlers may boost performance.

Conclusion

Seriously, if you made it this far, thank you for reading! There’s nothing like getting to share code with each other and improve the technologies and frameworks we get to use every day. If you see any issues or want to see a feature added, please open one on Github, I’ll be quick to respond to it!

Lastly, many, many more options are available for the configuration of your references if you read through the README .

https://github.com/deviantony/docker-elk

Enjoying these posts? Subscribe for more