Reuse the Code Climate schema when validating degradations

What does this MR do and why?

  • Gitlab::Ci::Reports::CodequalityReports#valid_degradation? built a new JSONSchemer schema for every degradation. Each build resolved the schema path on disk and re-read codeclimate.json.
  • Every degradation is validated twice, once in CodeClimate#parse_all and once in add_degradation. For gitlab-org/gitlab that is about 40,000 schema builds per merge request comparison.
  • This MR builds the schema once per thread (Thread.current[:codeclimate_schema]) and reuses it for every later degradation on that thread.
  • This is the main cost behind the slow ReactiveCachingWorker jobs that overspend the code_scanning error budget.

Results

Measured locally on GDK with the real gitlab-org/gitlab Code Quality reports from !257526 (merged) (19,988 findings across head and base). Five runs per method, medians shown.

before-after-cpu

Method CPU before CPU after Change Objects before Objects after Change
Harness (parse, compare, serialise) 7.53s 1.54s -80% 25.66M 5.72M -78%
In context, MergeRequest#calculate_reactive_cache 6.54s 1.55s -76% 24.54M 5.75M -77%
In context, real ReactiveCachingWorker job 7.01s 1.66s -76% 24.55M 5.76M -77%

The Sidekiq numbers come from the job's line in log/sidekiq.log, which has the same fields as production:

Sidekiq field Before After Production before (gitlab-org/gitlab)
cpu_s 7.01s 1.66s about 20 to 30s
mem_objects 24.55M 5.76M about 24.5M
mem_total_bytes 2.50 GB 0.33 GB about 2.85 GB
duration_s 7.61s 1.70s about 30s
  • ✅ Output is unchanged. The comparison result has the same SHA-256 digest in all 30 runs, before and after, across all three methods.
  • ✅ File.realpath (56% of CPU samples) and IO.read (17%) are gone from the CPU profile.
  • ✅ Cost still grows linearly with findings, now at about 0.07ms per finding instead of about 0.37ms.
  • Local allocations match production. Local CPU is about 3 to 4 times faster, so production jobs for gitlab-org/gitlab should drop from about 20 to 30s to about 5 to 7s. That is an estimate, not a production measurement.
Why the schema is cached per thread
  • json_schemer 2.4.0 lazily builds a keyword location tree on the schema during validate (lib/json_schemer/schema.rb:410, lib/json_schemer/location.rb:13). Caching per thread means no schema object is shared between threads, matching SarifSchemaValidator (ee/lib/gitlab/security/parsers/validators/sarif_schema_validator.rb:20).
  • Each cached schema holds about 24.5 KiB, so a process holds at most 24.5 KiB times the threads that validate a report. Repeated validation does not grow it. Details: !257662 (comment 3900886849)
  • If building the schema raises, nothing is cached, valid_degradation? returns false, and the next call retries. A spec covers this.
  • Measurements for the per-thread version were within noise of the results above, with identical output.

References

Screenshots or screen recordings

Not a UI change. See the chart and tables under Results.

How to set up and validate locally

  1. Run the new specs:

    bin/rspec spec/lib/gitlab/ci/reports/codequality_reports_spec.rb

    Before this change, the example reuses the parsed schema instead of building it for every degradation fails because JSONSchemer.schema is called for every degradation.

  2. To reproduce the measurements, download these scripts into tmp/codequality-repro/ (keep in_context/ as a subfolder), with the head and base reports from the pipelines of !257526 (merged) in head/ and base/:

    Then run each script on master and on this branch:

    LABEL=before VARIANT=baseline bin/rails runner tmp/codequality-repro/bench.rb
    bin/rails runner tmp/codequality-repro/in_context/seed.rb
    LABEL=before bin/rails runner tmp/codequality-repro/in_context/measure_direct.rb
    LABEL=before bin/rails runner tmp/codequality-repro/in_context/measure_sidekiq.rb

Out of scope

These came up during profiling and need separate decisions:

  • CodequalityReportsComparer#existing_errors is always empty, because web_url differs between head and base.
  • Degradations with the same fingerprint overwrite each other in add_degradation.
  • Each degradation is still validated twice. With the schema cached, that is about 0.3s of the remaining 1.5s.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Philip Cunningham

Merge request reports

Loading
Loading