Urgent Maintenance
Incident Report for Bottlenose
Postmortem

We would like to apologize for this database incident. It resulted in significant downtime for several of our sites and intermittent downtime for many sites.

The issue was caused by human error. A database query was run on Tuesday, May 9 at approximately 3:50 PM EST which inadvertently deleted many records from related tables. Some of the deleted records contained critical information causing some sites to not launch or be unavailable.

We were able to restore the missing data from backups. However, we had not trained for this type of restoration and we had not performed such a restoration before. This resulted in us not being able to restore the data as quickly as we would have liked. Going forward we have learned and implemented procedures which will result in a much quicker restoration should an incident like this arise again.

We have also implemented safeguards on the data which was inadvertently removed to minimize this type of error from happening in the future.

Sincerely, William Carr

Posted May 10, 2017 - 15:34 EDT

Resolved
Bottlenose has completed the required database maintenance. We apologize for this unforeseen incident.
Posted May 10, 2017 - 14:43 EDT
Identified
Bottlenose experienced an issue with our database server which is requiring urgent maintenance. During this time, there may be intermittent downtime while we restart the database server. We are actively attending to the situation.
Posted May 10, 2017 - 10:26 EDT