Email sending suspended

Incident Report for Bottlenose

Postmortem

We are sorry for the issue with email sending. We know outbound email is important for our customer’s businesses.

The issue was caused when one of our customer’s email account was compromised. A spammer used the account’s credentials to send spam. This resulted in our entire organization’s emails to be blocked from being sent via Amazon Web Services (AWS) Simple Email Service (SES). Bottlenose uses AWS SES for all upstream SMTP email. This meant no email marketing, transactional emails, or personal emails were able to be sent through Bottlenose.

We noticed an issue with a monitor we thought was in place to notify us of a situation like this. Had this monitor worked as expected we likely would not have had any issues. We have repaired the monitor.

We are also working with the AWS support team to see how we may improve our response. There were communication issues between Bottlenose and AWS support that resulted in the incident taking a much longer time to be resolved.

Posted Nov 08, 2022 - 09:04 EST

Resolved

AWS SES has unpaused our account for sending. Outbound (SMTP) email is now functioning as normal for email marketing, transactional email, and personal email.

Posted Nov 08, 2022 - 08:27 EST

Monitoring

We are still working with AWS to get our sending ban lifted.

As a workaround, while we get this issue resolved, we have made a new mail configuration that will allow mail to be sent.

Posted Nov 08, 2022 - 00:29 EST

Update

AWS support has escalated the issue. They say it may take up to 2 hours to get it resolved. We continue to monitor the issue.

Posted Nov 07, 2022 - 17:16 EST

Update

We have removed the offending email account from our system. We are still waiting for AWS to lift sending ban. We have a support case with AWS and are waiting for their reply.

Posted Nov 07, 2022 - 15:37 EST

Update

We are continuing to work on a fix for this issue.

Posted Nov 07, 2022 - 15:33 EST

Update

We are continuing to work on a fix for this issue.

Posted Nov 07, 2022 - 15:26 EST

Identified

Our upstream email service provider, Amazon Web Services (AWS) Simple Email Service (SES), has suspended our account for sending emails. Spammers used a compromised email account to send unwanted emails. We have disabled the email account and are working with AWS to restore sending.

Posted Nov 07, 2022 - 14:12 EST

This incident affected: Email (SMTP Email, Transactional Emails, Email Marketing, POP3/IMAP Email).