Geo: Help diagnose assets 404ing on secondary site due to GitLab version mismatch
I notice that the GitLab revision is different between primary and secondary sites.
Primary:
gitlab-rake gitlab:env:info
inkubectl exec -it gitlab-toolbox-866c5dcdf8-76fpg -- bash
:GitLab information Version: 15.6.0-pre Revision: 9a9664c0baf
Secondary:
sudo gitlab-rake gitlab:env:info
SSH'd intostaging-ref-3k-geo-gitlab-rails-3
:GitLab information Version: 15.6.0-pre Revision: 1516ec580b4
I suspect this is the reason for the CSS and JS assets 404ing. If the secondary is proxying most requests to the primary as expected, then the responses are using asset bundles named as they are on the primary. But those don't match the assets that the secondary has. And assets requests are not proxied.
If the asset 404s are fixed, then that may fix the partially blank pages.
The revision is displayed on each site on Admin > Geo > Sites
, and it is red on the secondary when it does not match the primary. This is helpful, but we could help people a bit more.
Ideas:
- Document the assets 404 on secondary sites in Geo Troubleshooting. Weight 0.5.
- Document that during GitLab upgrades, you should temporarily point the secondary domain to the primary (for separate URLs), or remove the secondary site from DNS (for Unified URL). To avoid routing users to the secondary site which will experience downtime of the UI due to assets 404s.
- Add a check to
rake gitlab:geo:check
. If revision does not match, then say e.g. "Geo does not support using multiple versions of GitLab at the same time. Assets requests will 404 on secondary sites.`. Weight 1 - Ideally: Add logic to Workhorse to proxy assets when "this secondary site" is on a different GitLab version than the primary. I think this could be like in the Geo proxy API endpoint adding a field in the response "proxy_assets: true/false". Workhorse can use this to conditionally remove the assets route from geoLocalRoutes. This solution makes a version mismatch cause a UI performance regression rather than a broken secondary UI. Weight 4