Reduce repeated Git commit lookups in CI job hooks

What does this MR do and why?

CI job hooks repeatedly fetch the same Git commit from Gitaly. This MR caches the commit ID, message, author name, and author email for 10 minutes behind the default-off ci_build_hook_commit_cache flag. Job and pipeline fields, pinned event state, and the current author URL remain specific to each event. The worker skips payload construction when no job hooks or integrations remain.

Cache design and consistency

The cache is keyed purely on immutable inputs: project ID, storage shard, repository disk path, and the content-addressed commit SHA. The MR adds no schema change, no invalidation writes, and no forced-primary reads.

A commit SHA is content-addressed, and the Gitaly RewriteHistory RPC runs git-filter-repo with only --strip-blobs-with-ids and --replace-text, which touch blob contents, never commit messages or author fields. Cached metadata for a SHA therefore matches what Gitaly returns for it for as long as the object exists, and no invalidation is needed for correctness.

Accepted residual, documented in Gitlab::Ci::BuildHookCommitCache: after RewriteHistory and a subsequent gc prune, an orphaned SHA stops resolving in Gitaly while a cached entry still would, for at most the 10-minute TTL. Such an entry exists only because an earlier delivery for the same SHA already fetched and sent the same metadata to the same recipients, so the residual difference is payload shape, not content.

The worker keeps its normal data_consistency :delayed behavior: Project.find_by_id remains a replica read, and history rewrite (Repositories::RewriteHistoryService) is unchanged.

Cache behavior and recovery
  • Keys include project ID, storage shard, repository disk path, and SHA.
  • Only the four commit fields are cached. Missing commits, invalid metadata, and field values above 64 KiB bypass caching. Redis errors use the original Gitaly lookup.
  • No invalidation exists: creation, destruction, housekeeping, and history rewrite cannot change the cached fields for an existing SHA. The accepted post-GC residual is documented in the cache class.
  • Concurrent cold misses can each call Gitaly. ObjectFormat RPCs are unchanged.
  • gitlab_ci_build_hook_commit_cache_reads_total counts hit/miss results. Bypasses are excluded; invalid entries and Redis read failures count as misses.
Validation and measured cost
  • Local RSpec on the changed spec files: 108 examples, 0 failures.
  • Coverage includes read-only/pending-delete/NULL-flag bypasses, cross-SHA and cross-project key isolation, missing and oversized metadata, Redis read/write failure fallback, and cache-metrics classification.
  • The onboarding request passes its original 101-query budget; this MR no longer touches that spec.
  • RuboCop checked all changed Ruby files with no offenses.

Three local repetitions used 20 same-SHA events across distinct jobs and two pipelines, with fresh RequestStore, BatchLoader, and query-cache contexts. Hook delivery was intercepted after recipient checks and real payload construction. Setup, invalidation between arms, and cache inspection were outside the measurements.

Measurement per 20 events Flag off Flag on
ListCommitsByOid RPCs 20 1
All Gitaly RPCs 40 21
ObjectFormat RPCs 20 20
Redis commands 440 271
SQL queries, including cached queries 400 400
Forced primary SELECTs 0 0

The RPC and Redis counts are from the recorded local experiment. The SQL and primary-SELECT rows are stated for the current design: the earlier generation lookup was exactly one primary SELECT per event, and removing it brings the flag-on SQL count to the flag-off value with no forced-primary reads. These samples establish fewer commit RPCs, not production latency.

The flag definition uses milestone 19.6. No backfill or new index is needed, and the MR adds no database migration.

References

Screenshots or screen recordings

Not applicable. This change affects background workers.

How to set up and validate locally

  1. Use a disposable project with several CI jobs and a job webhook connected to a local receiver.

  2. Select the project in the Rails console and disable the flag for the baseline:

    project = Project.find_by_full_path('<chosen-project-path>')
    Feature.disable(:ci_build_hook_commit_cache, project)
  3. Run a pipeline and record webhook build IDs, statuses, timestamps, pipeline fields, and commit metadata.

  4. Enable the flag and run another pipeline for the same SHA:

    Feature.enable(:ci_build_hook_commit_cache, project)
  5. Compare payloads and ListCommitsByOid calls in Gitaly logs. Event fields should remain distinct; ObjectFormat calls are unchanged.

  6. Disable job events on all project hooks and integrations, then produce another job event. Confirm that the worker constructs no job payload.

  7. Disable the flag after validation:

    Feature.disable(:ci_build_hook_commit_cache, project)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Aleksandr Kotlyar

Merge request reports

Loading
Loading