Skip to content

Race condition in commit status API creates multiple pipelines

Summary

Simultaneous requests to POST /projects/:id/statuses/:sha (https://docs.gitlab.com/ee/api/commits.html#post-the-build-status-to-a-commit) result in multiple pipelines being created.

Steps to reproduce

  1. Create a project, without .gitlab.ci.yml
  2. Push a new commit
  3. Post two distinct status updates to that commit at the same time e.g.

Request 1:

POST /projects/:id/statuses/:sha

{
  "name": "security",
  "status": "running",
  "description": "testing"
}

Request 2:

POST /projects/:id/statuses/:sha

{
  "name": "licenses",
  "status": "running",
  "description": "testing"
}

You should intermittently see the creation of two separate pipelines, rather than one pipeline.

We initially observed this behaviour during integration with Snyk. Snyk's integration with Gitlab posts two build statuses to Gitlab ("licenses" and "security").

Because the duplicate pipeline is created, subsequent updates (e.g. posting a success status) only apply to the latest pipeline, which leaves jobs in the second pipeline appearing to "run" forever.

What is the current bug behavior?

Simultaneous updates of commit status intermittently result in duplicate pipelines being created

What is the expected correct behavior?

Simultaneous updates of commit status should result in a single pipeline being created.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
sudo gitlab-rake gitlab:env:info


System information
System:         Ubuntu 16.04
Proxy:          no


Current User:   git
Using RVM:      no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown


GitLab information
Version:        12.0.3-ee
Revision:       1b1872f9d93
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.8
URL:            
HTTP Clone URL: 
SSH Clone URL:  
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml


GitLab Shell
Version:        9.3.0
Repository storage paths:
  • default: /data/git/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...


......(all green)..


Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Background

The relevant code seems to be this:

        pipeline = all_matching_pipelines.first

        ref = params[:ref]
        ref ||= pipeline&.ref
        ref ||= user_project.repository.branch_names_contains(commit.sha).first
        not_found! 'References for commit' unless ref

        name = params[:name] || params[:context] || 'default'

        pipeline ||= user_project.ci_pipelines.build(
          source: :external,
          sha: commit.sha,
          ref: ref,
          user: current_user,
          protected: user_project.protected_for?(ref))

        pipeline.ensure_project_iid!
        pipeline.save!

So if all_matching_pipelines doesn't return anything, a new pipeline is created.

        def all_matching_pipelines
          pipelines = user_project.ci_pipelines.newest_first(sha: commit.sha)
          pipelines = pipelines.for_ref(params[:ref]) if params[:ref]
          pipelines = pipelines.id_in(params[:pipeline_id]) if params[:pipeline_id]
          pipelines
        end

We do have params[:ref], so pipelines.for_ref(params[:ref]) is probably what doesn't find anything on the second API request if pipeline.save! hasn't yet finished for the first API request.

Possible fixes

Wrap the call in a redis-based exclusive lease when params[:pipeline_id] is not provided and an existing pipeline is not found. It would also be good to explicitly look up the existing pipeline from the primary database to avoid a race with database replication.

This is a moderately high traffic endpoint, so the change should be feature flagged.

The exclusive lease key would need to have a unique prefix and contain all of the following:

  1. user_project.id
  2. commit.sha
  3. params[:ref] when provided
Edited by Hordur Freyr Yngvason