Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • gitaly gitaly
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 557
    • Issues 557
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 57
    • Merge requests 57
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitalygitaly
  • Issues
  • #2951
Closed
Open
Created Jul 08, 2020 by Pavlo Strokov@8bitlifeMaintainer

Gitaly Cluster read distribution (General Availability)

Problem to solve

Repositories with many contributors combined with highly parallelized CI pipelines, and highly variable project access patterns can cause resource exhaustion, impacting availability. Mitigating these performance problems by sharding generally results significant over provisioning of CPU and memory to handle peak load.

Gitaly Cluster allows Git repositories to be replicated on multiple warm Gitaly nodes, and can be used to scale performance by distributing read operations between up-to-date replicas. When using Gitaly Cluster, from 13.3, read operations will automatically be distributed between up-to-date replicas. In comparison to three shards with one Gitaly node, a single Gitaly Cluster with three Gitaly nodes can share the available CPU and memory resources across all repositories.

Further details

There is a potential issue with performance gitlab-org/quality/performance#231 (closed) that needs to be handled before it can be marked as highly available.
The testing on gitlab.com is also blocked because of the !2340 (merged) as almost all replication events can't be properly processed, so there is no up-to-date secondaries that could be used for reads distribution. More info in the #2903 (closed).
It also requires visual presentation in Grafana dashboard to track it.

Proposal

  • Improve query performance and observe in production.
  • Enable read distribution gitaly_distributed_reads feature flag by default.
Edited Aug 05, 2020 by James Ramsay (ex-GitLab)
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking