We will be formalizing and streamlining our incident response process since some of the initial response to the issue could have been more prompt. Formalizing the process also makes it so that we don't skip steps (i.e. putting up a status page for the partial outage)
We are making changes to our observability platform to ensure that these types of issues are addressed in a more timely manner before customers bring it up.
Posted Jul 13, 2023 - 08:37 EDT
Resolved
Images edited in the builder before December 12th, 2022 used a service that would generate a URL based on the transformations applied to the images. Going forward, any new images edited will not use this service. As part of our security reviews, it was found that we needed to add trust hosts in order to validate that requests were being serviced from the right DNS name(s). Unfortunately, in tightening the restrictions and creating an allow list that included enterprise.knak.io, adding the edited-images.knak.io was missed and requests to this legacy service started failing.