Skip to content

Deployment order and concurrency

Description

Zendesk ticket: https://gitlab.zendesk.com/agent/tickets/37618

Note: This came up in the customer ticket above. However, I've also recently discussed this exact issue on IRC, too.

Consider this scenario. A project has a build and deploy process that deploys X to production after each commit to master. If multiple commits land in master within a short time of each other, two build/deploy pipelines will run at once. If the earlier commit's deploy finishes first, there is no problem. All fixes/features will end up in production after the two pipelines complete.

However, if the later pipeline finishes first, the earlier pipeline will deploy over top, effectively removing the later fix/feature from production. This could be a problem.

@markpundsack points out that likely the next build/deploy would push that commit to production again. Me: Eventually consistent? 😛

Potential problems

  • Where's the source of truth? GitLab probably shows that both pipelines completed, and deployed. The latest pipeline will appear to be the version deployed to production (or maybe not? We should verify this).
  • Could cause confusion. Developers or QA, or any other number of people see a complete pipeline and expect their fix/feature to be deployed. If X still doesn't work they could be confused in to thinking their 'fix' didn't actually fix the problem. Takes time to track down the issue - it didn't deploy now because another pipeline deployed over it.
  • Database/other data migrations could cause incompatibilities. If the later pipeline executes these migrations, then the code is rolled back by the earlier pipeline, the production application may break.
  • In cases of artifacts, images, etc., the pipeline could push to a repository and tag the image as the 'latest', which would not be true in all cases.

I'm not sure 'eventually consistent' is a great model for deployments, at least not a one-size fits all approach. My belief is that we'll see this topic come up as more people use GitLab for deployments. Many may not yet realize they have this problem but will probably notice it at some point.

  • What do other CI solutions do?
  • Do they account for this?
  • Do they have order/concurrency limits for deployments?
  • Is this really a big issue?

Proposal

  • Explicitly order and limit concurrency of deployment jobs.
  • Add an easy way for deployments to check their position. Am I deploying HEAD? If yes, go ahead. If no, skip the deploy.