Batch cached attribute reads for CiRunnerManager

What does this MR do and why?

Fields of GraphQL CiRunnerManager that read cached attributes (version, status, contactedAt, upgradeStatus and others) made one Redis GET for each manager on the page. A request that reaches 5,200 managers made 5,200 GETs, one after another, and took 4 to 10 seconds in Redis calls.

This MR reads the cached attributes of all managers on the page together:

  • RedisCacheable::CachedAttributesReader holds the cached attributes of one record and reads them from Redis at most once. CachedAttributesReader.preload fills many readers with one batched read. On a Redis Cluster it uses Gitlab::Redis::ClusterUtil.batch_get. On other Redis it uses MGET.
  • RedisCacheable.preload_cached_attributes preloads the readers of many records.
  • Types::Ci::RunnerManagerType and the EE upgradeStatus read cached attributes through with_cached_attributes. It uses a BatchLoader that preloads the managers that GraphQL returns. The batch covers only the managers on the page, not every manager that the resolver loaded.
  • The change is behind the gitlab_com_derisk feature flag ci_batch_runner_manager_cached_attributes, which is disabled by default. With the flag off, each manager is read as before.
  • with_cached_attributes yields the preloaded manager. A query can load the same manager twice, so that manager can be another instance of the same record.

On GitLab.com the keys are in different hash slots, so MGET is not possible. Redis still runs one GET for each manager, but the app sends them in a few pipelined batches instead of waiting for each reply. This lowers round trips and request time. It does not lower the number of Redis commands. The effect on Redis CPU is measured during rollout.

Not in this MR: RunnerManagersResolver still loads every manager of the requested runners from the database and slices the page in Ruby. The assessment tracks this separately.

Feature flag

  • Flag: ci_batch_runner_manager_cached_attributes (gitlab_com_derisk, disabled by default)
  • Rollout issue: #639071
  • Removal: config/feature_flags/gitlab_com_derisk/ci_batch_runner_manager_cached_attributes.patch removes the flag check and the specs for the disabled flag. The DeleteOldFeatureFlags keep applies it when it removes the flag.

References

Screenshots or screen recordings

Not applicable. There is no UI change.

How to set up and validate locally

  1. In a Rails console, create a runner with several managers and cache their attributes:
    runner = FactoryBot.create(:ci_runner)
    managers = FactoryBot.create_list(:ci_runner_machine, 5, runner: runner, version: '17.0.0')
    managers.each { |manager| manager.cache_attributes(version: '17.1.0') }
  2. In another terminal, watch the Redis commands: redis-cli -s $(gdk config get redis.__socket_file) monitor
  3. Run this query in GraphiQL at http://127.0.0.1:3000/-/graphql-explorer:
    { runners(first: 100) { nodes { managers(first: 100) { nodes { version status } } } } }
    You see one GET for each manager, and version is 17.1.0.
  4. Enable the flag in the console: Feature.enable(:ci_batch_runner_manager_cached_attributes)
  5. Run the query again. You see one MGET for all managers, and the same result.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marius Bobin

Merge request reports

Loading
Loading