PNGs and PDFs downloading as blank

Timeline

On October 8th, at 1:27PM EST we deployed a package upgrade to our backup rendering service which is responsible for generating previews of pdfs and pngs.

At 2:06PM EST we routed our production traffic to our backup rendering service. During which time it was monitored noting that the service was succeeding users were able to download pdfs and pngs

At 3:01PM EST Knak Engineers were informed that the pdfs and pngs being download were blank pdfs and pngs. At 3:18PM EST Knak Engineers reverted the traffic back to the primary rendering service, which they then confirmed that pdfs and pngs were correctly downloading

Root Cause

An underlying package that our rendering service relies on has implemented a breaking change. It was believed that the package upgrade followed standard semantic versioning which would mean that this was minor change which would not impact production, it was later discovered it does not follow semantic versioning. This was not noticed in our non production environments as functionally from an integration standpoint our rendering service was taking in code and rendering a pdf/png.

Action Taken

• We have rolled back the change and went back to testing the package upgrade
• We have documented how to manually end to end test our rendering service
• We are working on implementing a fully automated end to end test for our application/rendering service