Email Services Interruption

Incident Report for Bottlenose

Postmortem

I would like to apologize for the recent outage with our email delivery system. We realize email marketing, transaction emails, and email sending and receiving are important to our customer’s business. We take very seriously providing a consistent and reliable email delivery platform.

Bottlenose uses Amazon Web Services (AWS) Simple Email Service (SES) for upstream email delivery. At 11:53 AM EST on November 14 we received a notice via email from AWS about an immediate ban on Bottlenose's access to SES. This ban had an immediate impact on all email services including email campaign delivery and testing, transactional emails, and email sending and receiving. In over five years of using this service, we had never received such a ban.

We worked with AWS support engineers to identify the reason for the ban which was a dramatic increase in spam coming from Bottlenose's email system as determined by AWS's spam filtering system. AWS identified six user accounts with Bottlenose as the source of the increased spam. Our internal investigation revealed these accounts had likely been hacked and were being used to send spam unbeknownst to the customers with these accounts. We changed the passwords for these accounts and applied to the SES team to have the ban lifted.

We placed an alternative outbound SMTP transport in place to handle outbound email until SES could be restored. Unfortunately, this transport was not as reliable as SES and many upstream recipients such as @comcast.net, and yahoo.com blocked email from this transport. While some emails were being sent though others were not.

After further phone calls to the AWS premium support team, the ban was lifted around 10 AM on November 15. We cleaned our email delivery queue of any suspicious emails in order to not be banned again and restored the outbound AWS SES transport. At this point, we began to see our email related services return to normal.

We have made some changes to our monitoring system which will help us avoid a situation like this in the future. We recognized a pattern in our server logs which is a good indicator of this type of compromise. We have installed a monitor and alert to let our technical staff know when an email account has been compromised using this pattern. We believe we will be able to stop this type of abuse before it gets to the point of getting us banned from SES.

Sincerely,
William Carr
President
Bottlenose

Posted Nov 15, 2019 - 14:25 EST

Resolved

It looks like all email services have returned to normal after restoring AWS Simple Email Service. We have been monitoring our mail system for the last several hours and all systems appear to be functioning as normal.

Posted Nov 15, 2019 - 13:57 EST

Monitoring

AWS SES mail transport has been enabled. Our mail queue deliveries are proceeding without issue. We believe this issue is resolved and normal email deliveries should occur. We expect email campaign delivery, email campaign testing, transactional email sending, email reveiving, and email sending to function as normal. We will keep this incident open and continue to monitor.

Posted Nov 15, 2019 - 10:44 EST

Update

AWS SES has lifted Bottlenose's ban. This is good news. However, before we re enable sending though AWS we need to be sure we do not get banned again. We have about 44,000 messages in our mail queue awaiting delivery. We will be working with AWS support to ensure when we send these messages though it wil not result in another ban. Once we are certain we will not get immediately banned we will renable email sending though AWS.

Posted Nov 15, 2019 - 09:43 EST

Update

We are continuing to work on a fix for this issue.

Posted Nov 14, 2019 - 22:00 EST

Update

AWS SES has blocked all sending from Bottlenose due to a dramatic increase in levels of spam. We have identified about 6 user accounts on the Bottlenose platform responsible for the spam messages. We believe these accounts have been compromised. We are changing the login credentials for each of these accounts and applying to AWS SES to have our ban lifted.

In the meantime, we have removed AWS SES sending from our email platform. This means email sending may proceed. However, the reason Bottlenose uses AWS SES is to acheive a higher level of email deliveries and we may not have the same level of deliveries until we are able to reinstate AWS SES. We consdier the current situation less than optimal and we will continue to purse a resolution.

Posted Nov 14, 2019 - 19:16 EST

Identified

There is an issue with our connection to our service provider AWS Simple Email Service. We are working with their support now to get the issue resolved.

Posted Nov 14, 2019 - 16:34 EST

Investigating

We are currently investigating reports of email bouncing back. We are working with our service provider to determine the source of the issue.

Posted Nov 14, 2019 - 13:53 EST

This incident affected: Email (SMTP Email, Transactional Emails, Email Marketing, POP3/IMAP Email).