We started noticing slowness and outages with Platform 1 websites during the morning of March 1. There had not been any major changes to Platform 1 for months, only routine system upgrades. This was a perplexing issue. We threw more power at the problem by increasing both the number of instances and the power of the instance classes. This measures had some positive effect but the problem persisted.

After analyzing server logs we noticed frequent and unusual requests from an ip address originating in the United Arab Emirates. Blocking this ip address and its associate neighbors had an immediate effect. Server load went down instantly and response times returned to normal.

We are now running with our normal amount of instances for Platform 1 but we have chosen to stay with the higher instance class.