2024-06-05: page styling broken in staging-canary causing all QA tests to fail
Customer Impact
Deployment blocked at staging-canary
, no direct customer impact. NOT occurring in staging
Current Status
- all smoke tests failed in staging-canary (example failure issue: https://gitlab.com/gitlab-org/gitlab/-/issues/465589)
- First pipeline with failing tests https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/pipelines/3350712
- First deployer pipeline that resulted in failing tests https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/3350173
- This is not a MIME type issue, the problem is that when you access Staging canary one of the asset files required is not present in the asset bucket on Staging https://staging.gitlab.com/assets/tailwind-e500910b5189045b8f20487439479185b059a6d74411d347e087f19b215ce4fb.css
- You can reproduce this issue if you set your canary cookie on next.gitlab.com and go to https://staging.gitlab.com/help
- Canary staging is running 17.1.202406050201-56e36cab013.72d88a18dd4
- The deployer pipeline for this is https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/3351343
- For this release we uploaded assets to the bucket in this job https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/14190843 with
gitlab-assets-ee:56e36cab01359c392e3aca896f28c9ab69daeace
Steps to reproduce:
Switch staging to Next
and load:
https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2024-06-04-23-44-07-f58d5c44943323a4/project-with-merge-request-03ab18af807fb7dc
Errors present in console:
Console errors: Refused to apply style from 'https://staging.gitlab.com/assets/tailwind-ba024276344884dac7b15b9a330bafe9a95b0bc5e882aa06c37c15b57d432996.css' because its MIME type ('application/xml') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
📚 References and helpful links
Recent Events (available internally only):
- Feature Flag Log - Chatops to toggle Feature Flags Documentation
- Infrastructure Configurations
- GCP Events (e.g. host failure)
Deployment Guidance
- Deployments Log | Gitlab.com Latest Updates
- Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook
Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:
- Corrective action ❙ Infradev
- Incident Review ❙ Infra investigation followup
- Confidential Support contact ❙ QA investigation
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.
Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.