Email marketing outage
Incident Report for Bottlenose
Postmortem

I want to apologize for the issue with the email marketing campaigns yesterday on platform 2. We know that the timely delivery of email campaigns is vital to our customers' businesses.

Our email marketing platform relies on a public website for "CSS inlining" of the email messages. CSS inlining refers to a technique for formatting the messages to have the correct styling in web-based email clients such as Gmail.
We think sometime yesterday, the public website we relied on changed their interface. Our email marketing platform could not communicate with it anymore to get the email messages formatted. This prevented any messages from being sent, including test messages.

Our development team worked to correct the issue but could not make it work with the previous method. We attempted writing our own custom CSS inliner code, but the task proved too daunting to be completed quickly. Finally, we identified another public website providing the CSS inliner service provided by Mail Chimp. We integrated their inliner into our email marketing platform and deployed it into production around 0:30 EST. At this point, we observed normal email campaign deliveries and campaign testing.

To prevent an issue like this from happening again, we are planning the development of a new CSS inliner. We may develop the inliner in-house or identify a supported third-party service.

Sincerely,
William Carr

Posted Oct 27, 2020 - 08:52 EDT

Resolved
We installed a fix for the email campaign service. This fix uses a different method for inlining the CSS necessary for formatting emails for web-based email clients like Gmail. We are monitoring the changes and so far we are seeing the email marketing system running as normal.
Posted Oct 27, 2020 - 00:51 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 27, 2020 - 00:42 EDT
Identified
A third-party online resource Bottlenose's email marketing platform uses for creating emails has stopped responding. This tool is called a CSS Inliner and is used to "inline" the styles used in the marketing emails. CSS inlining is necessary to have emails display with expected colors and styles in many online email clients including Gmail.

We are working on finding an alternative CSS inliner as our top priority. As soon as the fix is in email campaigns will resume sending.
Posted Oct 26, 2020 - 17:03 EDT
This incident affected: Email (Email Marketing).