Skip to content
GitLab
Next
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 44,761
    • Issues 44,761
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,330
    • Merge requests 1,330
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • GitLabGitLab
  • Issues
  • #36154
Closed
Open
Issue created Nov 12, 2019 by Stan Hu@stanhuOwner

Reduce contention on updating runner_id in ci_builds

From https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8395#note_243107996:

On the master, UPDATE ci_builds .. were taking 3/4 of all total_time today, see K003 for the master: image

This becoming more and more noticeable: every second, 8 out of 12 seconds spent by the master's CPUs were doing this work.

It's a contention problem - queries are blocked by some other queries that work on the same records. Note how low shared_blks_hit and shared_blks_read are, we don't have a lot of data to process, we just wait being blocked. It leads to having more and more Postgres backends blocked and sitting in active state. We should avoid this.

Can we rework it to use SELECT .. FOR UPDATE SKIP LOCKED or SELECT .. FOR UPDATE NOWAIT? This would completely solve the problem of waiting being blocked. Similar work was done recently for merge_requests by @stanhu, see !18481 (diffs).

This SQL query is likely happening in assign_runner!: https://gitlab.com/gitlab-org/gitlab/blob/febfd21ee6733f11b9713ab7ee968bae190c10ef/app%2Fservices%2Fci%2Fregister_job_service.rb#L95-116

Assignee
Assign to
Time tracking