Sending e-mails on a schedule with Sendgrid and Firebase

Job Dispatch is a personalized jobs newsletter that sends you new jobs from companies you follow every week. It's a tool I've been building together with @bzaalig over the last 3 months.

I wanted to write down the experience developing Job Dispatch using Webflow, Nodejs, Firestore, Firebase functions and storage, Sendgrid and a few 3rd party APIs like Lever and Greenhouse.

My goal writing these posts is to give some insight into how actual projects get built, rather then this becoming a technical tutorial to setting up Firebase and Sendgrid.

Onward!

What we're making

Job Dispatch is a newsletter of sorts. So it was inevitable that at some point I had to think about scheduling e-mails. That being said, most SaaS need some form of e-mail scheduling! Think about confirmation e-mails or perhaps an onboarding drip campaign.

At Job Dispatch, we send out a few different scheduled e-mails :

• A feedback email four days after a user received their first Dispatch
• A reminder e-mail to confirm their e-mail address after 1 day
• An onboarding e-mail with how to use the product a few hours after they confirmed their e-mail address.

Setting up Firestore & Data structure

I have three collections set up for our emails: templates, scheduledEmails , andemails. Like the name implies, scheduledEmails represent e-mails to be sent at some point in the future.

Emails is the collection that is hooked up to the firebase-sendgrid extension — but more on that later. Templates is where I store the HTML template for each email type.

Here's how a single scheduledEmail will look like:

{
  "type": "WelcomeEmail",
  "ScheduledDate": Firestore.Timestamp,
  "to": "hello@toonverbeek.com",
  "html": ""
}

And here's a template:

"welcomeEmail": {
    "subject": "Welcome to Job Dispatch!"
    "html": "Hi {{firstName}} ..."
}

Note that the document ID is not randomly generated. I created the documents manually and gave them the ID equal to the email type (e.g: welcomeEmail). Also note the {{firstName}}, which indicates that this html is a handlebars template, allowing us to interpolate variables. More on that when we configure Sendgrid!

Next, let's take a look at setting up Firebase functions.

Sending e-mails with Firebase background functions

Our scheduled e-mails will always get created by a trigger. For example: the welcomeEmail is triggered after a user confirms their e-mail address. And the howTo email is created a day after a user confirms their e-mail address. So in this case, both e-mails rely on the same trigger: a user confirming their e-mail.

In Firebase, this can be achieved by using a background trigger and a scheduled function. You can find the entire script that takes care of scheduling e-mails at the end of the post. For now, let's go over it step by step. Start by adding this to your empty index.js:

//index.js

const functions = require('firebase-functions');
const admin = require('firebase-admin');

const { isPast, addHours } = require('date-fns');

admin.initializeApp();

// https://firebase.google.com/docs/functions/tips#use_global_variables_to_reuse_objects_in_future_invocations
const db = admin.firestore();

After initializing Firebase, we're going to start by exporting two functions. Each exported function will be deployed in seperate containers and scaled up and down as necessary by the Firebase platform.

exports.onUserUpdate = admin.firestore.document('users/{uid}').onUpdate(handleCreateScheduledEmails);
exports.sendScheduledEmails = functions.pubsub.schedule('every 60 minutes').onRun(handleSendScheduled

The onUserUpdate function is triggered whenever a document in the users collection is updated and will take care of creating any e-mails that should be scheduled. The sendScheduledEmails gets called every 60 minutes and simply polls a collection of scheduled emails to check if any new e-mails should be sent.

Creating the scheduled e-mails

Next, let's create those two handler functions. Let's start by creating the handleCreateScheduledEmails function in index.js:

async function handleCreateScheduledEmails(snap, context) {
    const { confirmed, emailAddress } = snap.data().after;
    const wasPreviouslyConfirmed = snap.data().before.confirmed;
    const isUserConfirming = confirmed && !wasPreviouslyConfirmed

    // 1. Check if the user is confirming their e-mail address
    if (!isUserConfirming) return null;

    // 2. Create welcomeEmail
    const welcomeEmail = createWelcomeEmail(emailAddress);
    // 3. Create howTo email
    const howToEmail = createHowToEmail(emailAddress);

    // 4. Return a single promise that resolves if all provided promises also resolve
    return Promise.all([welcomeEmail, howToEmail]);
}

This function takes care of creating the welcome and onboarding email after a user confirms their e-mail address. Every time a user document gets updated, this function will run. That's why we include a check to see whether the new data coming in is actually an update to the confirmed property of the user.

To create the e-mails we have two simple functions, createWelcomeEmail and createHowToEmail. These return promises, which we resolve by returning Promise.all(). We'll get to those later.

Sending the scheduled e-mails

Now that we've taken care of creating scheduled e-mails, let's write the logic for sending them. Add the sendScheduledEmails function to index.js:

async function handleSendScheduledEmails(snap, context) {
    const scheduledEmailsResult = await getScheduledEmails();
    const promises = [];

    scheduledEmailsResult.forEach((emailDoc) => {
        const email = emailDoc.data();
        const { scheduledDate, sendgridEmail, status } = email;

        // .toDate() is needed because this is a Firebase Timestamp
        if (isPast(scheduledDate.toDate()) && status === 'PENDING') {
            promises.push(sendEmail(sendgridEmail, emailDoc.ref));
        }
    });

    // We're using .allSettled() because the results are not dependent on each other
    // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled
    const promiseResults = await Promise.allSettled(promises);
    for (const result of promiseResults) {
        if (result.status === 'rejected') {
            functions.logger.error(`Could not send scheduled email: `, result.reason);
        }
    }
    return null;
}

This function gets executed every 60 minutes, as specified by our exported pubsub function. Each time it will check if there are any e-mails ready to be scheduled by iterating over the scheduledEmails collection and for each document checkign if the scheduledDate is in the past. We import the isPast function from date-fns to help us do this. If it is, we queue an e-mail by calling the sendEmail and adding it to a promises array.

I find handling promises like this a convenient way to combine functional programming (i.e: using .forEach()) and still neatly queueing promises.

Lastly, we await each promise using Promise.allSettled and log an error message if something went wrong.

All that rests is implementing the other helper functions. You can add them to your index.js:

function createWelcomeEmail(emailAddress) {
    const today = admin.firestore.Timestamp.now();
    const scheduledDate = addHours(today.toDate(), 4);

    const emailConfig = {
        to: emailAddress,
        type: 'welcomeEmail',
        scheduledDate
    }

    return createScheduledEmail(emailConfig);
}

function createHowToEmail(emailAddress) {
    const today = admin.firestore.Timestamp.now();
    const scheduledDate = addHours(today.toDate(), 24);
    const data = { firstName: "Bob" }
    const emailConfig = {
        to: emailAddress,
        type: 'howToEmail',
        data,
        scheduledDate
    }

    return createScheduledEmail(emailConfig);
}

async function createScheduledEmail({ to, type, data, scheduledDate }) {
    const scheduledEmail = {
        scheduledDate,
        status: 'PENDING', // PENDING, COMPLETED, ERROR
        sendgridEmail: {
            to,
            template: { name: type, data }
        }
    };

    return db.collection('scheduledEmails').add(scheduledEmail);
}

async function getScheduledEmails() {
    return db.collection('scheduledEmails')
        .where('scheduledDate', '<=', admin.firestore.Timestamp.now())
        .where('status', '==', 'PENDING')
        .orderBy('scheduledDate')
        .get();
}

async function sendEmail(email, scheduledEmailRef) {
    try {
        await db.collection('emails').add(email);
        return scheduledEmailRef.update({ status: 'COMPLETED' });
    } catch (error) {
        await scheduledEmailRef.update({ status: 'ERROR', error: error.message });
        return Promise.reject(error.message);
    }
}

We can now move on to connecting our our e-mail provider, Sendgrid, to Firebase.

Using Sendgrid as a Firebase Extension

Extensions documentation

Extensions are like plug-ins for the Firebase ecosystem. They help developers ship apps faster by providing certain logic, like the ability to send e-mails, without writing code. We'll be using the Sendgrid extension so that we don't have to work with the Sendgrid API ourselves!

Configuring the extension is fairly straightforward. The SMTP Connection URI is your smtps://apikey:<your_api_key>@smtp.sengrid.net:465. You'll need to setup a user and API key in Sendgrid first to get the key. email will be the collection that the extension will be listening for new emails to be sent. templates is where our templates are stored ;-).

Entire code for this post

Next up: storing HTML in Firebase Storage

Coming soon!