Optimize CI/CD component fetching with caching

Furkan Ayhanrequested to merge
584405-optimize-component-fetch-service-by-cache into master

What does this MR do and why?

Cache component content using Rails.cache to improve performance when projects use multiple CI/CD component includes. This reduces redundant component fetches during pipeline creation.

Changes:

  • Add Rails.cache for component content keyed by project path, SHA, and component name
  • Refactor sha resolution to use find_catalog_version directly

The cache uses SHA in the key, making it safe for long TTL since SHA content is immutable.

Behind feature flag ci_optimize_component_fetching #586889.

References

Related to #584405

Screenshots or screen recordings 

# RAILS_PROFILE=true GITALY_DISABLE_REQUEST_LIMITS=true rc

require 'benchmark'

ActiveRecord::Base.logger = nil
project = Project.find_by_full_path('root/basic')
user = project.first_owner

n = 10

# warm up
Ci::CreatePipelineService
  .new(project, user, ref: project.default_branch)
  .execute(:push)
  .payload

Benchmark.bm do |benchmark|
  benchmark.report("old") do
    Gitlab::SafeRequestStore.ensure_request_store do
      Thread.current[:ci_optimize_component_fetching] = false
      n.times do
        pipeline = Ci::CreatePipelineService
          .new(project, user, ref: project.default_branch)
          .execute(:push)
          .payload

        raise "stop!" unless pipeline.created_successfully?
      end
    end
  end

  benchmark.report("new") do
    Gitlab::SafeRequestStore.ensure_request_store do
      Thread.current[:ci_optimize_component_fetching] = true
      n.times do
        pipeline = Ci::CreatePipelineService
          .new(project, user, ref: project.default_branch)
          .execute(:push)
          .payload

        raise "stop!" unless pipeline.created_successfully?
      end
    end
  end
end

(n * 2).times do
  ::Ci::DestroyPipelineService.new(project, user).execute(Ci::Pipeline.last)
end

#          user     system      total        real
# old  1.181649   0.104702   1.286351 (  2.503468)
# new  1.367305   0.086884   1.454189 (  1.876191)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports