## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/634314) currently behind the `ci_archive_traces_cron_cursor` feature flag. - DRI: @mfanGitLab - Team Slack channel: `#g_pipeline-execution` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? Changes how `Ci::ArchiveTracesCronWorker` pages through stale traces. If it misbehaves, stale traces aren't archived (the same as today, where the cron times out every run). Watch the worker's duration and `json.extra.ci_archive_traces_cron_worker.result` in Sidekiq logs, and statement timeouts on the CI database. Since https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263377, the flag also changes how every job log is archived, including from `Ci::ArchiveTraceWorker`. A trace whose chunk data is missing is archived straight away with a marker in place of each lost chunk, and builds out of attempts get an incomplete archive. If this misbehaves, logs could be archived with markers that shouldn't have them. Turning the flag off reverts both changes. Watch: - `gitlab_ci_trace_archival_total` by `result`, especially `incomplete` and `failed`; - `Archived incomplete trace.` in Sidekiq logs; - Sentry for errors from `Ci::ArchiveTraceService`. Introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263376 and https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263377. ## Events - [Exceptions with ci_archive_traces_cron_cursor:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22ci_archive_traces_cron_cursor:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with ci_archive_traces_cron_cursor:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22ci_archive_traces_cron_cursor:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_pipeline-execution`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set ci_archive_traces_cron_cursor 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set ci_archive_traces_cron_cursor true --dev --pre --staging --staging-ref ``` **Production** — the flag uses the `:instance` actor, because the cron is instance-wide: ``` /chatops gitlab run feature set ci_archive_traces_cron_cursor true ``` The cron runs at minute 17 of every hour. Check the next run's result in Sidekiq logs before considering it stable. ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete ci_archive_traces_cron_cursor --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set ci_archive_traces_cron_cursor false # production /chatops gitlab run feature set ci_archive_traces_cron_cursor false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete ci_archive_traces_cron_cursor --dev --pre --staging --staging-ref --production # remove entirely ```