[FF] `job_analytics_siphon` -- Read CI job analytics from siphon_p_ci_builds

Summary

Roll out the feature currently behind the job_analytics_siphon feature flag.

When enabled, Ci::JobAnalytics::QueryBuilder reads finished CI builds from the Siphon CDC replica (siphon_p_ci_builds) via ClickHouse::Finders::Ci::SiphonFinishedBuildsFinder, instead of the ci_finished_builds-backed FinishedBuildsFinder/FinishedBuildsDeduplicatedFinder. The flag is project-scoped.

DRI: @narendran-kannan
Team Slack channel: #g_ci_platform

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

Read-only path: this only changes which ClickHouse tables the job analytics read path queries; no writes, no data-loss risk.
Metric drift: the siphon finder derives duration from age('ms', started_at, finished_at) (no duration column on siphon_p_ci_builds) and excludes timestampless builds via NULL, matching the legacy finder. Watch for unexpected differences in mean_duration/percentiles between flag-on and flag-off.
Siphon replication lag/gaps: if siphon_p_ci_* replication is behind, analytics may show fewer/stale rows than the materialized-view path. Compare row counts during rollout.
Dashboards to watch on https://dashboards.gitlab.net: ClickHouse query latency/errors and the Verify/CI analytics GraphQL error rate.

Rollout

Run all production /chatops in #production and cross-post the results to #g_ci_platform. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set job_analytics_siphon 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set job_analytics_siphon true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set job_analytics_siphon <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss job_analytics_siphon true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com job_analytics_siphon true
/chatops gitlab run feature set --user=narendran-kannan job_analytics_siphon true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Docs + version history updated
Breaking changes announced, if any
Change management issue opened, if required
External API consumers handled with a fail-open mechanism, if applicable

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete job_analytics_siphon --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set job_analytics_siphon false                                         # production
/chatops gitlab run feature set job_analytics_siphon false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete job_analytics_siphon --dev --pre --staging --staging-ref --production  # remove entirely

[FF] job_analytics_siphon -- Read CI job analytics from siphon_p_ci_builds