[FF] ci_optimize_artifact_parsing -- optimize CI artifact parsing and size validation

Summary

Roll out optimized CI artifact parsing currently behind the ci_optimize_artifact_parsing feature flag.

  • DRI: @hfyngvason
  • Team Slack channel: #g_pipeline-execution

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The flag changes how remote (object storage) artifacts are downloaded for parsing in Ci::Artifactable#each_blob (JUnit reports, dotenv, annotations): one download to a tempfile reused for size validation and parsing, instead of one download for validation plus a chunked re-download for parsing. Validation semantics are unchanged.

If the new path fails, artifact parsing raises for jobs finishing while the flag is enabled (for example, missing test report results), with no data loss elsewhere. Rollback is disabling the flag. Watch Sidekiq error rates for Ci::BuildFinishedWorker and Gitlab::Ci::Artifacts::DecompressedArtifactSizeValidator::FileDecompressionError occurrences in Sentry/Kibana.

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline-execution. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set ci_optimize_artifact_parsing 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set ci_optimize_artifact_parsing true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set ci_optimize_artifact_parsing <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_optimize_artifact_parsing true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com ci_optimize_artifact_parsing true
/chatops gitlab run feature set --user=hfyngvason ci_optimize_artifact_parsing true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/244619 19.2
/chatops gitlab run feature delete ci_optimize_artifact_parsing --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set ci_optimize_artifact_parsing false                                         # production
/chatops gitlab run feature set ci_optimize_artifact_parsing false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete ci_optimize_artifact_parsing --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Hordur Freyr Yngvason