[FF] initial_pipeline_process_worker_ci_primary_reads -- read the CI database from the primary in Ci::InitialPipelineProcessWorker

Summary

Roll out the change currently behind the initial_pipeline_process_worker_ci_primary_reads feature flag.

When the flag is enabled, Ci::InitialPipelineProcessWorker reads the CI database from the primary while main database reads stay sticky, the same effective setting PipelineProcessWorker has. When the flag is disabled, the worker keeps today's data_consistency :sticky for every database. The motivating bug is https://gitlab.com/gitlab-org/gitlab/-/issues/632778 (confidential).

  • DRI: @tigerwnz
  • Team Slack channel: #g_pipeline-execution

Note

Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability. It is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The worker runs once per pipeline creation. With the flag on, its CI database reads move from replicas to the primary, so the blast radius is extra read load and connection use on the CI primary. The expected impact is small: the worker already switches to the primary as soon as it writes, which it does on almost every run, and PipelineProcessWorker has read CI from the primary since 2021 at a higher rate.

Watch while rolling out:

No data is written differently with the flag on or off. Rollback is immediate: the next job evaluates the flag again.

Events

Feature flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline-execution. Background: incremental rollout process, feature actors.

The flag is checked with Feature.current_request as the actor inside the Sidekiq server middleware, so percentage-of-actors rollout applies per job. Project, group and user actors do not apply to this flag.

Non-production

  • /chatops gitlab run feature set initial_pipeline_process_worker_ci_primary_reads 50 --actors --dev --pre --staging --staging-ref
  • /chatops gitlab run feature set initial_pipeline_process_worker_ci_primary_reads true --dev --pre --staging --staging-ref

Production, percentage rollout (wait at least 15 minutes between steps, watch the dashboards above):

  • /chatops gitlab run feature set initial_pipeline_process_worker_ci_primary_reads <percentage> --actors
  • /chatops gitlab run feature set initial_pipeline_process_worker_ci_primary_reads true

Before global rollout

Confirm the relevant gotchas before going to 100%. See enabling a feature for GitLab.com.

  • Docs and version history: not applicable, no user-facing behaviour changes.
  • Breaking changes: none.
  • Change management issue: open one if required by the process for the CI primary.
  • External API consumers: not applicable.

Cleanup

Remove the flag once deemed stable. See cleaning up. The cleanup merge request replaces the flag-gated override in Ci::InitialPipelineProcessWorker with data_consistency :always, overrides: { main: :sticky }, deletes the YAML definition, and carries the Changelog: fixed trailer for https://gitlab.com/gitlab-org/gitlab/-/issues/632778. Then:

  • /chatops gitlab run release check <merge-request-url> <milestone>
  • /chatops gitlab run feature delete initial_pipeline_process_worker_ci_primary_reads --dev --pre --staging --staging-ref --production

Rollback

  • Production: /chatops gitlab run feature set initial_pipeline_process_worker_ci_primary_reads false
  • Non-production: /chatops gitlab run feature set initial_pipeline_process_worker_ci_primary_reads false --dev --pre --staging --staging-ref
  • Remove entirely: /chatops gitlab run feature delete initial_pipeline_process_worker_ci_primary_reads --dev --pre --staging --staging-ref --production