Don't use the application as the backend for assets
Right now, we have CDNs configured for GitLab.com, staging, and canary, using the GITLAB_CDN_HOST env var.
These work by using the application itself as a backend for a Fastly CDN. And that is (at least, in part) why we have three different CDNs - staging, canary, and production can all be running different versions of the application, with different assets. The application package itself contains the bundled CSS, JS, etc.
For GitLab.com usage, we don't actually need to use the application package for the assets at all, though. Instead, we could:
- When a release is tagged (could be on other conditions), run an extra CI job.
- This job would push the assets to object storage after they're built. (They will be included in packages by the other CI jobs).
- Use that object storage as the CDN's backend for assets, rather than the application itself.
As far as I'm aware, we also use the CDN for assets that aren't bundled with the application - like user uploads and avatars - so we would have to be able to switch backend based on the path (starts with /assets/, I think).
This would prevent issues like https://gitlab.com/gitlab-com/infrastructure/issues/4879.