Deployment of 11.9 RC6 to canary caused 401 and 404s when loading emojis
After 11.9.0 RC6 was deployed to canary, some users on GitLab.com started seeing Something went wrong while loading emojis
. @skarbek drained canary traffic from HAProxy until we sort this out.
- Slack thread: https://gitlab.slack.com/archives/C101F3796/p1552509890553000
- Related issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/58784
Summary:
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26102 removed the use of CDN from loading the emojis.
- When the application attempts to request emojis and store them in local storage, it will use
https://gitlab.com/-/emojis/1/emojis.json
. - This will be a 404 to a logged-in user because traffic gets routed directly to the production Web workers. Having a CDN host would at least make the request go to the canary backend, but now that endpoint is going directly to the 11.9.0 RC3 workers.
This makes a commonly-cached file hit the main Web server.
Edited by Stan Hu