Take control of your backend with Firebase Cloud Functions (II)

Use Firebase Realtime Database to implement an easy yet powerful API cache for your mobile apps.

In the previous post, we learned how to use Firebase Cloud Functions to clean up a backend response and make it more mobile friendly.

In this post, we are going to show how to use Firebase Realtime Database to save that cleaned response, using it as a cache. This will prevent calling the original backend API too frequently as well as unnecessary transformations.

Firebase caching the transformed response from our backend

Why use a cache at all?

The cloud function we developed in part one was used to fetch raw data from our backend and transform it into something easier to process by a mobile app.
But, in the majority of cases, there’s no need to fetch new fresh data every time a client requests it: cheaper, cached data is good enough.

What to cache and for how long will depend on several factors unique to your case:

• Business rules: perhaps your backend only produces new data at certain known times. Some newspapers, for example, only have morning, midday and evening editions. Or maybe you know your backend only generates updated data every hour.
• Costs: putting together the data a mobile client is requesting could be an expensive operation for your backend.
• Time: this is related to the previous point. If generating requested data is expensive, then it is most certainly going to be slow. And you don’t want to waste users’ time waiting.

In all these cases it seems wise to establish a data validity policy and rely on already cleaned cache data instead of fetching the raw data for every request.

As an added bonus, having our cached data in Firebase allows us to share cache logic across multiple clients.

Persist cleaned up model to Firebase

Continuing with the example introduced in part one, what we are going to save is the cleaned up feed from The Guardian. The feed, once cleaned looks like this:

Cleaned up feed from The Guardian

There are three things to consider for this example:

1. Saving cleaned up data into our project’s Firebase database.
2. Checking if there is valid cached data before fetching new.
3. Cache invalidation policy: deciding when this cache is not valid anymore.

1. Saving transformed data into Firebase database

In this gist, you can find the full code for fetching and transforming data we did in part one.

Let’s start by refactoring that code into something more Promising

exports.fetchGuardian = functions.https.onRequest((req, res) => {
    return request(URL_THE_GUARDIAN)
        .then(data => cleanUp(data))
        .then(items => response(res, items, 201))
});
function request(url) {
    return new Promise(function (fulfill, reject) {
        client.get(url, function (data, response) {
            fulfill(data)
        })
    })
}
function response(res, items, code) {
    return Promise.resolve(res.status(code)
        .type('application / json')
        .send(items))
}

This code is equivalent to the one in the previous post, but by using Promises, the flow is easier to understand and modify.

I’m omitting here the cleanUp function as it’s not relevant, but check the previous post if you are interested in it.

The first thing we are going to do is saving items in Firebase database before returning them to the client. That can be done by modifying the previous code and adding a call to save(items) in the Promises chain:

exports.fetchGuardian = functions.https.onRequest((req, res) => {
    return request(URL_THE_GUARDIAN)
        .then(data => cleanUp(data))
        .then(items => save(items))
        .then(items => response(res, items, 201))
});
function save(items) {
    return admin.database().ref('/feed/guardian')
        .set({ items: items })
        .then(() => {
            return Promise.resolve(items);
        })
}

With admin.database().ref('feed/guardian') we obtain a reference to a path in our database.
Then, set({items: items}) pushes the items array to that path in the database with the key "items" and returns an empty Promise.

Finally, when the promise is fulfilled (meaning the writing process is done) we return a new Promise with the original items array to continue the chain.

In Firebase this will generate:

Feed persisted in our Firebase Realtime Database

2. Checking cached data before fetching new

At this point, we are persisting the cleaned up data in our database but we are not doing anything with it.

The next step is to check if there’s saved data in Firebase database before making an HTTP request to our backend.

exports.fetchGuardian = functions.https.onRequest((req, res) => {
    return admin.database().ref('/feed/guardian')
        .once('value')
        .then(snapshot => {
            if (isCacheValid(snapshot)) {
                return response(res, snapshot.val(), 200)
            } else {
                return request(URL_THE_GUARDIAN)
                    .then(data => cleanUp(data))
                    .then(items => save(items))
                    .then(items => response(res, items, 201))
            }
        })
});
function isCacheValid(snapshot) {
    return (snapshot.exists())
}

What changes from step 1 is that we read from the database before fetching the feed.

As we already know, admin.database().ref('feed/guardian') is a path in the database. With .once('value') we read the values at that path once and return a Promise with them.

What we do afterward is simple:

• If the data read from /feed/guardian is valid, we return it (at this point valid means just existing)
• If the data is not valid (it doesn’t exist) we do exactly the same thing we were doing in step 1: read from the original feed, persist in our Firebase database and return.

3. Cache invalidation policy

Finally, we need to set rules for cache validity. To keep this example simple we are going to consider the data is valid for 1h since it’s saved. After that time, the next request should fetch new items and replace the cached ones with them.

To do that we need to save the fetching time along with the items when we persist them. We need to modify our save function like this:

function save(items) {
    return admin.database().ref('/feed/guardian')
        .set({
            date: new Date(Date.now()).toISOString(),
            items: items
        })
        .then(() => {
            return Promise.resolve(items);
        })
}

This will produce a new field in our database:

New date field

The last bit is to use this date to check how old our cached data is.

If the cached data was saved less than one hour ago we’ll consider it valid. Otherwise, we invalidate the cache by fetching fresh data and overriding it.

function isCacheValid(snapshot) {
    return (
        snapshot.exists() &&
        elapsed(snapshot.val().date) < ONE_HOUR
    )
}
function elapsed(date) {
    const then = new Date(date)
    const now = new Date(Date.now())
    return now.getTime() — then.getTime()
}

All together

The code is also available here as a gist snippet

Full code for this example

Stay tuned!

This is part two of a three-part series of articles about Firebase.

In the final instalment we’ll learn how to use Google Cloud Natural Language API from our Firebase Cloud Function to enrich the backend response, for example, adding sentiment analysis.

— -

Find me on Twitter @lgvalle, I’d love to chat about Android, Firebase & Cloud Functions.