Skip to content

Fix performance of the pipelines JSON endpoint

Furkan Ayhan requested to merge fa/fix-performance-for-pipelines-endpoint into master

What does this MR do and why?

The origin of this performance problem is the pipelines tab of merge requests. When loading the pipelines, we send a request to "merge_requests/:id/pipelines" and return a JSON payload.

Previous improvement: !128097 (merged).


This JSON payload contains all "failed_builds". However, we don't need this calculated array because we only need the number of failed builds. In this MR, we are removing the failed_builds field from the pipelines serializer.

Benchmark shows that this change improves the performance by 59%.


This change is behind the feature flag ci_fix_performance_pipelines_json_endpoint. (#427255 (closed))

Screenshots or screen recordings

With a merge request that has many pipelines;

Screenshot_2023-09-29_at_15.52.52

Have this change to benchmark;

expose :failed_builds, if: -> (_, options) { !options[:ci_fix_performance_pipelines_json_endpoint] && can_retry? }, using: Ci::JobEntity do |pipeline|
  pipeline.failed_builds.each do |build|
    build.project = pipeline.project
  end
end

Run the console with RAILS_PROFILE=true rails c.

Run the benchmark;

require 'benchmark/ips'

user = User.find(1); nil
project = Project.find_by_full_path('root/gitlab-replica'); nil
merge_request = MergeRequest.find(105); nil
pipelines = Ci::PipelinesForMergeRequestFinder.new(merge_request, user).execute; nil

ActiveRecord::Base.logger = nil

::Gitlab::SafeRequestStore.ensure_request_store do
  # warmup
  PipelineSerializer.new(project: project, current_user: user).represent(pipelines, preload: true)
  PipelineSerializer.new(project: project, current_user: user).represent(pipelines, preload: true, ci_fix_performance_pipelines_json_endpoint: true)

  Benchmark.ips do |x|
    x.warmup = 0
    x.time = 20

    x.report "without FF" do
      PipelineSerializer.new(project: project, current_user: user).represent(pipelines, preload: true)
    end

    x.report "with FF" do
      PipelineSerializer.new(project: project, current_user: user).represent(pipelines, preload: true, ci_fix_performance_pipelines_json_endpoint: true)
    end

    x.compare!
  end; nil
end

result;

Calculating -------------------------------------
          without FF      1.723  (± 0.0%) i/s -     34.000  in  20.157212s
             with FF      2.731  (± 0.0%) i/s -     54.000  in  20.210030s

Comparison:
             with FF:        2.7 i/s
          without FF:        1.7 i/s - 1.59x  slower

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports