Failed to create resources during deployment
Summary
Having a .gitlab-ci.yaml
with 1 stage (deploy) and 2 concurrent jobs (staging and production), sometimes both pass. Sometimes one passes and the other fails with the message:
This job failed because the necessary resources were not successfully created
Steps to reproduce
Disclaimer 1: Unfortunately, one might need to try all this steps several times since this is an intermittent problem. By simply rerunning the pipeline multiple times I was not able to reproduce it. I really had to create new clusters.
Disclaimer 2: For quickly creating clusters I suggest using this tool: https://gitlab.com/DylanGriffith/gitlab-cluster-helper
Disclaimer 3: I suggest disabling shared runners and only use one runner installed through GitLab Managed Apps
- Clone this project:
- Install a K8s cluster
- Install Helm
- Install Runner
- Disable Shared Runners
- Install Knative (domain name can be example.com)
- Run a pipeline.
Example Project
https://gitlab.com/Alexand/concurrent-deploys
What is the current bug behavior?
One of the concurrent deployment jobs fails sometimes.
What is the expected correct behavior?
They should both always pass.
Relevant logs and/or screenshots
Output of checks
This bug happens on GitLab.com
Possible fixes
@DylanGriffith raised the concern that we only create the resources(Service Acc, Namespace) needed for the deploy once, if these aren't already created. So, there could be a race condition between this two concurrent jobs.
I might not be describing the issue correctly, so please feel free to point out if this doesn't make sense as a possible cause.