We would like to apologize for this database incident. It resulted in significant downtime for several of our sites and intermittent downtime for many sites.
The issue was caused by human error. A database query was run on Tuesday, May 9 at approximately 3:50 PM EST which inadvertently deleted many records from related tables. Some of the deleted records contained critical information causing some sites to not launch or be unavailable.
We were able to restore the missing data from backups. However, we had not trained for this type of restoration and we had not performed such a restoration before. This resulted in us not being able to restore the data as quickly as we would have liked. Going forward we have learned and implemented procedures which will result in a much quicker restoration should an incident like this arise again.
We have also implemented safeguards on the data which was inadvertently removed to minimize this type of error from happening in the future.
Sincerely, William Carr