On October 26 and 27 we experienced several periods of slow response times and outages with our web services. These were caused by much higher than normal CPU load on our web server instances. There were three contributors to the increased server load; a misconfigured web crawler, a major increase in calls to our store API service, and a software misconfiguration. We have addressed each of these issues and now are seeing much better response times and availability.
I apologize for these issues.
Sincerely,
William Carr
Posted Oct 28, 2021 - 00:16 EDT
Resolved
This incident has been resolved.
Posted Oct 27, 2021 - 23:56 EDT
Monitoring
We have identified what looks like a misconfigured or nefarious robot that was sending a large amount of unwanted traffic to our servers. The traffic overwhelmed our servers causing slow response times and major availability issues. We have blocked the robot and response times are returning to normal.
We will continue to investigate.
Posted Oct 27, 2021 - 16:59 EDT
Investigating
We are currently investigating a service outage. The outage is affecting all consumer-facing and admin portal web services.
Posted Oct 27, 2021 - 16:37 EDT
This incident affected: eCommerce (Homepage, Site Search, API, Admin Dashboard) and Management Portal, Site Search.