Starting at 455PM EST, on January 18th, 2021, we saw a sudden spike in requests that caused our application instance to auto-scale. Unfortunately, because of the sudden nature of the spike, the auto-scaling did not occur fast enough to meet the demand for requests. Within moments, we scaled from less than 200 requests per min, to nearly 6000. This resulted in some issues for approximately 10 mins during the initial spike.
The issue was traced back to an email containing a specific transformed image, causing a spike in requests on our end.
Although the vast majority of our images are served behind a Content Delivery Network (CDN) through AWS, the edited images are currently served through the application directly. Of the efforts to load the image, 238,852 were successful, and 24,873 failed.
We have made changes to our infrastructure in order to serve our edited images through a CDN in much the same way as our uploaded images are currently, in order to avoid this type of issue in the future.