Retry job token auth on primary after stale status read

What does this MR do and why?

A job token is only handed to the runner after the job transitions to running (the token is rendered in the POST /jobs/request response, after the transition commits). Observing the job in a pre-execution status (created, waiting_for_resource, preparing, pending) during job token authentication is therefore only possible when the job was read from a stale replica. Today this raises NotRunningJobError, the endpoint returns 403, and the runner terminates the job.

This MR adds a retry in Ci::AuthJobFinder#validate_executing_job!: when a pre-execution status is observed, pin the CI database session to the primary (use_primary!), re-read the job once, and re-check. The primary pin covers the remainder of the request, so subsequent reads (for example trace chunk state) are also served fresh. If the job is still not executing on the primary, the error is raised as before.

Design notes:

• The retry is cause-agnostic: it heals the empty Redis sticking-key window (30s TTL vs. up to ~60–240s of tolerated replica lag before health checks evict a host), Redis eviction/failover, and any residual pre-commit-LSN sticking not covered by the ci_stick_build_after_commit rollout (!230249 (merged)).
• The trigger is anti-correlated with benign cases: it only fires when a stale read is provable from the status itself, so the primary cost is one PK point-read per genuine incident — strictly less than the existing use_primary_on_failure sticking fallback that already moves whole request sessions to the primary during lag.
• Placing the retry in Ci::AuthJobFinder (rather than the runner API helper) covers every job-token-authenticated surface: runner job endpoints, package registry, dependency proxy, and API calls using CI_JOB_TOKEN.
• The retry is logged (json.message: job token auth retried on primary after stale status read) with a job_recovered field, so we can measure residual staleness after the ci_stick_build_after_commit rollout and decide whether further work (for example raising the sticking TTL) is warranted.

Gated behind the default-disabled gitlab_com_derisk feature flag ci_job_token_auth_stale_read_retry, checked per project.

References

• Fixes (residual cases of) #596961
• Complements !230249 (merged) (ci_stick_build_after_commit)
• Rollout issue: #605475

How to set up and validate locally

1. Enable the feature flag:

   Feature.enable(:ci_job_token_auth_stale_read_retry, Project.find(<project_id>))

2. Simulate a stale replica read by putting a running job's status back to pending with the token still valid:

   job = Ci::Build.find(<job_id>) # a running job
   token = job.token
   job.update_column(:status, :pending)

3. Call a job-token-authenticated endpoint (for example GET /api/v4/job with JOB-TOKEN: <token>). Without the flag the request fails with 403; with the flag it is retried on the primary — since the primary also shows pending in this simulation, it still fails, but the retry is visible in log/application_json.log with recovered: false.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.