Mail System Response Times
Incident Report for Bottlenose

I’d like to apologize for the response times and outages with our mail systems over the last few days.

We first recognized the problem when a large amount of spam email was causing slow response times on our mail servers. There was such a large amount of spam messages coming in that our message scanners were becoming overwhelmed and could not keep up with the demand. What made matters worse was the email appeared legitimate on many of the preliminary checks. The messages were coming from Google’s mail servers and passed DKIM and SPF checks.

We temporarily blocked an entire class B network from Google on Friday evening. This stopped the spam attack but prevented any legitimate email originating from Google from being accepted by our mail systems. When we removed the temporary Google block on Friday night we noticed mail servers operating at normal loads. We thought the attack had stopped and everything way okay. However, we discovered on Monday, a misconfiguration in one of the mail servers caused mail to not be received and sent throughout the weekend. The misconfiguration was introduced while we were trying to find the block for the spam attack.

We now have the spam attack blocked in a way which will not compromise mail system response times. We are working with Google’s abuse prevention team to stop the attack at its source. It is continuing but we are blocking it much more efficiently now.

I know email is important to your businesses and I apologize for any problems this incident has caused.


Bill Carr

Posted Sep 09, 2019 - 16:31 EDT

We are still receiving tremendous amounts of email from a spammer using Google's mail servers. We have found a way to block the spam farther upstream which will keep our mail servers running at desired performance levels. We continue to work through Google's abuse reporting channels to stop the email at its source.
Posted Sep 09, 2019 - 14:55 EDT
We are receiving a tremendous amount of spam addressed to users on our email platform. The large volume of spam is overwhelming our spam filtering systems and creating overall slow response times in all mail systems. We are working at cutting off the spam from its upstream source.
Posted Sep 09, 2019 - 10:36 EDT
This incident affected: Email (SMTP Email, Transactional Emails, Email Marketing, POP3/IMAP Email).