<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Collaborate/take over this issue](https://contributors.gitlab.com/manage-issue?action=work&projectId=278964&issueIid=596974) </details> <!--IssueSummary end--> ## Summary This issue is to roll out the CI component includes caching feature on production, that is currently behind the `ci_cache_component_includes` feature flag. **Feature flag type:** `ops` This flag gates Redis-backed (`Redis::RepositoryCache`) cross-request content caching for CI/CD component includes (`include:component`). When enabled, component file content fetched from Gitaly during pipeline creation is cached using SHA-keyed cache keys with a 4-hour TTL. Since Git SHAs are immutable, the cache is inherently safe and requires no invalidation logic. **Note:** This flag is intended to remain in the codebase long-term. It will be enabled at 100% on GitLab.com but **not** rolled out to self-managed, since we cannot predict the Redis capacity of self-managed deployments. This allows quick toggling if Redis memory saturation occurs. This flag was renamed from `ci_optimize_component_fetching` (type `gitlab_com_derisk`) to `ci_cache_component_includes` (type `ops`) to match the naming convention of `ci_cache_project_includes`. Since the previous flag was already at 100% on GitLab.com, this rollout will go directly to 100%. Introduced in: !231756 Related: #588313 ## Owners - Most appropriate Slack channel to reach out to: `#g_pipeline-authoring` - Best individual to reach out to: @avielle ## Expectations ### What are we expecting to happen? When enabled, CI pipeline creation will cache the content of `include:component` files in `Redis::RepositoryCache` with a 4-hour TTL. Subsequent pipeline creations that reference the same component at the same SHA will read from Redis instead of making Gitaly calls. This should: - Significantly reduce Gitaly load from CI component file fetching - Reduce intermittent `TimeoutError` failures for customers with large numbers of includes - Decrease pipeline creation latency for pipelines that use common CI/CD components ### What can go wrong and how would we detect it? 1. **Redis memory saturation**: Caching file content in Redis increases memory usage. Monitor Redis memory metrics on `Redis::RepositoryCache`. If memory usage spikes, disable the flag immediately. - Dashboard: https://dashboards.gitlab.net (Redis RepositoryCache metrics) 2. **Stale content served**: This should not happen because cache keys include the Git SHA (immutable), but if somehow stale content is observed, disabling the flag will bypass the cache entirely. 3. **Increased error rates on pipeline creation**: Monitor Sidekiq and web error rates for `Ci::CreatePipelineService`. If error rates increase after enablement, disable the flag. ## Rollout Steps Since the previous flag (`ci_optimize_component_fetching`) was already enabled at 100% on GitLab.com, this is a direct rollout after the rename MR is deployed. ### Rollout on non-production environments - [ ] Enable the feature globally on non-production environments with `/chatops gitlab run feature set ci_cache_component_includes true --dev --pre --staging --staging-ref` ### Rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] Verify the rename MR !231756 has been deployed to production - [ ] Enable globally: `/chatops gitlab run feature set ci_cache_component_includes true` - [ ] Delete the old flag: `/chatops gitlab run feature delete ci_optimize_component_fetching --dev --pre --staging --staging-ref --production` ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops gitlab run feature set ci_cache_component_includes false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops gitlab run feature set ci_cache_component_includes false --dev --pre --staging --staging-ref ```