Improve Go build caching strategy in CI

What does this MR do?

Modifies the Go build cache strategy used in CI by:

  1. Removing the -$CI_COMMIT_REF_PROTECTED and -false suffixes from the cache location and directory to be cached, respectively.
  2. Keying the cache on "${CI_JOB_NAME}" instead of "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}".

Why was this MR needed?

Rationale behind the changes above:

  1. The original approach was used to isolate the caches used for non-protected branches from protected branches, and was added before GitLab had the CI/CD setting "Use separate caches for protected branches". It is redundant now that the setting is enabled for the project. Also, this approach meant that protected branches (e.g., main, release branches) didn't benefit from caching at all.
  2. The per-branch key approach meant that every new branch's first pipeline built from scratch.

Some notes on the new approach:

  • Protected branches now benefit from caching.
  • The first pipeline runs on MRs now benefit from caching.
  • The cache is split into two scopes, protected and non-protected, with each keyed per job.
  • Obsolete build artifacts (e.g., from now-removed dependencies or old Go versions) should be automatically removed over time by Go's built-in cleanup mechanism for cache entries that have not been used in 5 days.
  • Go's build cache is content-addressed: an entry is only reused when its full inputs (Go toolchain, build tags, source/dependency content) hash identically. So sharing one cache across branches can't cause incorrect reuse (mismatched entries are simply ignored).

Measured speedup (validated on this MR by re-running each job):

Job Cold Warm Reduction
binaries 54m 40s 10m 59s ~80% (−43.7m)
unit test 17m 51s 3m 55s ~78% (−14.0m)
go:lint 6m 11s 3m 15s ~47% (−2.9m)

Notably, binaries is on the critical path, so its warm start shortens the wall-clock time for the full pipeline.

What's the best way to test this MR?

  • Inspect the cache behaviour and runtime of the binaries, unit test, and go:lint CI jobs on their initial (cold) runs and second (warm) runs.
  • Post-merge checks:
    • main's first pipeline after merge should come up warm, since it shares the "protected" cache pool seeded by the jobs in this MR (the protected pool is used because I'm a Maintainer who triggered the pipeline).
    • Once the "non-protected" cache pool is seeded, all future pipelines in each scope benefit from the Go build cache.
    • Monitor the archive_cache size (it should self-limit rather than grow unbounded).

What are the relevant issue numbers?

N/A

Edited by Daniel Keenan

Merge request reports

Loading