How we do email at Makeshift
Our approach to the bit of the app no one wants to work on
For better or worse, email is really important across all aspects of a digital business. Done correctly it helps with acquisition (email marketing), activation (drip/on-boarding email), retention (behavioural email) and referral (invite your friends!) by bringing customers back to your product at the right time. In addition, many aspects of a digital product simply need email to function at all (transactional email such as ‘forgot your password’).
In this post I’m going to explore why (and which) email is important at Makeshift, and then outline our current email architecture. Hopefully it will be useful for your business if you’re also battling with email optimisation. I’d love to get your feedback on twitter - I’m @choosenick.
With the death of RSS and the rise of closed social platforms, email is one of the last places to get free distribution, and it’s almost totally universal. People spend ages in their email, and, as all direct marketers know, good email marketing cuts through.
So it’s fair to say that email really is one of the most central components of the internet operating system. Perhaps the most important component. Indeed, it’s even been argued that all software naturally tends towards the characteristics of email as it grows.
At Makeshift this is certainly true. Although we have a team of software people who all dislike email (much like other software companies), the reality is that a lot of our apps are effectively front ends to email.
LinkyDink is, literally, a method for creating relevant emails with links in for your team. Hire my Friend is simply a UI that helps decide who should see who’s email address. And many people who use Attending do it primarily because it helps them capture email addresses of people going to their events.
The problem is that email is hard. It’s hard to design, hard to build, hard to test. And because it’s hard, it often gets neglected by our product designers because, frankly, it’s more fun designing UI features that work in the browser.
Our original email stack was, to say the least a muddle, and born out of a series of small decisions taken in the moment, that were right for that situation, but had caused us to build up significant technical debt. In addition, because at Makeshift we work on multiple products simultaneously, we had a variety of email architectures operating across our various services.
Given that our entire commercial hypothesis rests on the assumption that we can generate efficiencies in learning and execution by working on multiple things at once, and that all our apps use email in a deep way, it seemed sensible to take stock, review, and re-design.
So last month we decided to tackle this and create an email working group looking at this problem across all our apps. We realised that email service design is a process, not a single project. It will never be perfect, and we need to allow room for innovation and change.
When we started this email review process we had a very heterogenous system, with no single design patterns. We were using Intercom, plus Mandrill plus some ActionMailer, plus Mailchimp for delivery, and our reporting relied on, well, we weren’t really sure. Quite a lot of bits and bobs, but no one source of truth.
The reason we had various email service providers providing various email services is that email exists as a kind of ghost version of all our apps, covering a very wide range of use cases and existing in various forms in almost all the stages of the user journey. So as we’d been building the product we’d been making (sort of) local design decisions in the same way we do with the core website design.
When we took a step back for a high level view of all our emails, we realised that emails exist on a spectrum. This goes from 100% transactional email to 100% marketing email, and, annoyingly, individual messages often defy specific categorisation.
The challenge here is, in part, that different people are responsible for different emails – transactional emails that contain lots of variables and relate to performing actions that connect deeply with the operation of the service are the responsibility of developers to build, whereas newsletters are the responsibility of marketing focused people.
You don’t want to deploy your app to send an email newsletter. But equally, when you send a newsletter that sends customers back to the app you want it recorded against their user ID.
Another additional complexity is the link into ‘Customer Relationship Management’. Email can often be used as a unique identifier for customers, and we have multiple apps, so when we want to contact customers we want one place where the conversation is recorded. Oh, and it’s the help desk. That’s an email thing too, and because Intercom is very clever and brings together behavioural email + helpdesk + CRM-ish features we had all that mixed up.
So, we set to work reviewing providers, searching for information on how other people are solving this problem, and then we sat down and discussed it as a team. Below I’m going to outline the resulting design principles, and where we’ve got to in terms of transactional email, and where we want to get to to in terms of behavioural and marketing email in the next few months.
These are our current principles. They are not all the principles, and we hope to refine and improve them over time, but the overall goals of our email architecture are:
Reliable and testable
We want the email parts of our software to perform to exactly the same standards as all the other parts. And that means we need to be able to test it, be confident in it, and be able to generate reports about it quickly and easily.
One source of truth in Postgres
We run all our apps on Postgres, and we do most of our reporting via our own analytics tool called Wrangler. Wrangler allows us to write reports across multiple Postgres databases with single SQL queries (yes, you read that right - multiple databases queried by one SQL report), so we always want all our data available in a Postgres DB.
Never have anything to do with delivery
We don’t want to have to deliver email! So we’ll always use someone bigger and better to do the job for us.
Make it easy to change/test the content of emails
We want to be able to change the content of emails as easily as possible
There are going to be more. Once we get our combined transactional / behavioural / marketing email stack going I’m sure we will add to this.
Design and management
To design the first versions of our emails we use a mixture of tools. We have a single spreadsheet where we keep a list of all the emails, and describe who they are for, what conditions they should be sent under and what the goal of the email is.
Then, once we know we want to send an email we produce a wireframe version, either using a drawing tool like Omnigraffle or Sketch or by just jumping into HTML. It isn’t always that easy to prototype emails in the browser because of the table based layouts you have to use for HTML email. We use the Zurb Ink framework to build our HTML emails.
Once an email is designed, we then integrate it into our app, and it is sent using the architecture below
Sending a transactional email
When a user performs an action that triggers a transactional email we first create a record in the Emails table of the whole email, including it’s unique ID. This is then sent via the Mailer Controller to a job queue, where jobs are then pulled and the HTML email and metadata is sent to our email sending app (currently Mailgun). This process is outlined below.
Reporting a transactional email
Mailgun then delivers the email, and checks for open and click results from the user. When it receives data back, this app then sends a webhook containing the email_ID as a POST back to our app, which sends a 200 ‘ok’ message back to Mailgun. We then write a record of the delivery, open, clicks and other meta data into a table called Email Logs in our main application DB.
So far so simple. With this architecture we get good test coverage and solid reporting, but at the expense of ease of management and change in the content of emails - we have to deploy the app to make changes that users will see. We’re ok with that, because the type of transactional email we are sending is deeply integrated into our services.
Within this transactional architecture we still have a big problem, that we’re currently looking to fix. If we draw the behavioural and marketing email sections of our architecture on the diagram we get this - a totally separate system, unconnected to the existing system.
Which is not good - and it’s actually worse because we actually have two services outside our ‘one source of truth in Postgres’ principle - Mailchimp and Intercom (which sends a lot of our email right now). Both of these services store the click and open data, and the customer data in their own databases with limited access. Certainly not a Postgres DB that we control.
So, going forward we are exploring several options, all focused on the idea of abstracting our email sending into a separate app, which can support 3rd party front ends for creating emails (e.g customer.io) and multiple back ends (e.g our products Hire my Friend + Attending + Wrangler etc)
This could allow for several useful things. First, it puts us further towards ‘one single source of truth in Postgres’ as all our emails for all our apps across the full email spectrum are going through one app (although with Wrangler multi data sources this is achievable with foreign data wrappers).
Secondly, we can then create and optimise a separate Emails database and increase performance - It’s not very performant to be adding an individual entry for every individual email sent to the Emails table in each app, especially once you start to include newsletter type emails sent to 10,000 people that are all exactly the same.
Hopefully this has been useful. As you can tell by the slight trailing off at the end of this post we’re still working on it - here’s our latest email update on Hire my Friend.
We’ve learnt a lot over the past month or so by concentrating on this problem, not least that email service design is a process in exactly the same way that our core product design work is a process.
As we find out more we’ll post another update here – if you want to be notified when that happens, add you email to the box below.
Thanks for reading! Nick - @choosenick