2023-11-16: Pipelines using service>variables are failing on GitLab.com
Customer Impact
Services defined in CI pipelines are not having environment variables passed in.
Potential workaround: Specify the variables in the global section: https://docs.gitlab.com/ee/ci/services/#passing-cicd-variables-to-services
(or)
Passing the env variables directly to the shell and execute the program in docker: gitlab-org/gitlab#431980 (closed)
Current Status
- We identified the root-cause to be gitlab-org/gitlab!112907 (merged), which was deployed on the production
2023-11-15 15:50 UTC
- We got first report in public issue in gitlab-org/gitlab#431980 (closed), which has also gotten some comments.
-
We have opened a revert for this MR in gitlab-org/gitlab!137084 (merged) which is already merged and hasGiven the time it would take for deploy to happen, we've decided to do a roll-back instead.~Pick into auto-deploy
applied and ETA for deployment is ~5 hours, so MR is likely to land in .com approximately by2023-11-16 11:30 UTC
. - So far we have received 3 customer tickets for this.
📝 Summary for CMOC notice / Exec summary:
- Customer Impact: Services defined in CI pipelines are not having environment variables passed in.
- Service Impact: devopsverify grouppipeline authoring
- Impact Duration: 05:03 UTC - 07:55 UTC (approx. 2 hours 52 minutes)
- Root cause: gitlab-org/gitlab!112907 (merged)
📚 References and helpful links
Recent Events (available internally only):
- Feature Flag Log - Chatops to toggle Feature Flags Documentation
- Infrastructure Configurations
- GCP Events (e.g. host failure)
Deployment Guidance
- Deployments Log | Gitlab.com Latest Updates
- Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook
Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:
- Corrective action ❙ Infradev
- Incident Review ❙ Infra investigation followup
- Confidential Support contact ❙ QA investigation
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.