Deploys of new GitLab versions cause 404s and other errors with static assets
- Slack conversation: https://gitlab.slack.com/archives/C101F3796/p1555005200317400
After we attempted to deploy 11.10.0 RC6 to canary, users on canary experienced several issues:
Missing icons
The missing icons problem happens because the canary Web workers request https://gitlab.com/assets/icons-b2073af7ca0641787eb4596951b695079519da1937a262a886bb12c2b0fac181.svg
, but 11.9.7 only has https://gitlab.com/assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg
.
Apparently this can't be solved by using the canary CDN because the browser disallows cross-origin requests (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5253#note_150448890).
Errors loading Worker objects
Similar to the icons, these Worker objects can't be loaded via CDN due to cross-origin restrictions (https://github.com/w3c/ServiceWorker/issues/940). Adjusting CORS rules do NOT help. Chrome doesn't use them for this.
Broken functionality due to missing assets
As mentioned in https://gitlab.com/gitlab-org/gitlab-ee/issues/11118, missing assets such as https://gitlab.com/assets/webpack/249.c227daf4.chunk.js
also break functionality. Batch reviews did not work quite right.
Possible solutions
- Proxy everything in
/assets
behind a CDN/bucket that contains all assets from previous and next releases - Support sticky load balancer sessions (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5253#note_150443444 ).
- Populate the Web worker machines with the required assets: https://ops.gitlab.net/gitlab-com/gl-infra/patcher/merge_requests/77, https://ops.gitlab.net/gitlab-com/gl-infra/patcher/merge_requests/78.
- Do a full deploy of the fleet immediately.