Reuse the Code Climate schema when validating degradations
What does this MR do and why?
Gitlab::Ci::Reports::CodequalityReports#valid_degradation?built a newJSONSchemerschema for every degradation. Each build resolved the schema path on disk and re-readcodeclimate.json.- Every degradation is validated twice, once in
CodeClimate#parse_alland once inadd_degradation. For gitlab-org/gitlab that is about 40,000 schema builds per merge request comparison. - This MR builds the schema once per thread (
Thread.current[:codeclimate_schema]) and reuses it for every later degradation on that thread. - This is the main cost behind the slow
ReactiveCachingWorkerjobs that overspend thecode_scanningerror budget.
Results
Measured locally on GDK with the real gitlab-org/gitlab Code Quality reports from !257526 (merged) (19,988 findings across head and base). Five runs per method, medians shown.
| Method | CPU before | CPU after | Change | Objects before | Objects after | Change |
|---|---|---|---|---|---|---|
| Harness (parse, compare, serialise) | 7.53s | 1.54s | -80% | 25.66M | 5.72M | -78% |
In context, MergeRequest#calculate_reactive_cache |
6.54s | 1.55s | -76% | 24.54M | 5.75M | -77% |
In context, real ReactiveCachingWorker job |
7.01s | 1.66s | -76% | 24.55M | 5.76M | -77% |
The Sidekiq numbers come from the job's line in log/sidekiq.log, which has the same fields as production:
| Sidekiq field | Before | After | Production before (gitlab-org/gitlab) |
|---|---|---|---|
cpu_s |
7.01s | 1.66s | about 20 to 30s |
mem_objects |
24.55M | 5.76M | about 24.5M |
mem_total_bytes |
2.50 GB | 0.33 GB | about 2.85 GB |
duration_s |
7.61s | 1.70s | about 30s |
✅ Output is unchanged. The comparison result has the same SHA-256 digest in all 30 runs, before and after, across all three methods.✅ File.realpath(56% of CPU samples) andIO.read(17%) are gone from the CPU profile.✅ Cost still grows linearly with findings, now at about 0.07ms per finding instead of about 0.37ms.- Local allocations match production. Local CPU is about 3 to 4 times faster, so production jobs for gitlab-org/gitlab should drop from about 20 to 30s to about 5 to 7s. That is an estimate, not a production measurement.
Why the schema is cached per thread
- json_schemer 2.4.0 lazily builds a keyword location tree on the schema during
validate(lib/json_schemer/schema.rb:410,lib/json_schemer/location.rb:13). Caching per thread means no schema object is shared between threads, matchingSarifSchemaValidator(ee/lib/gitlab/security/parsers/validators/sarif_schema_validator.rb:20). - Each cached schema holds about 24.5 KiB, so a process holds at most 24.5 KiB times the threads that validate a report. Repeated validation does not grow it. Details: !257662 (comment 3900886849)
- If building the schema raises, nothing is cached,
valid_degradation?returnsfalse, and the next call retries. A spec covers this. - Measurements for the per-thread version were within noise of the results above, with identical output.
References
- Error budget issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/628938
- Profiling that found the cause: https://gitlab.com/gitlab-org/gitlab/-/work_items/628938#note_3899581091
Screenshots or screen recordings
Not a UI change. See the chart and tables under Results.
How to set up and validate locally
-
Run the new specs:
bin/rspec spec/lib/gitlab/ci/reports/codequality_reports_spec.rbBefore this change, the example
reuses the parsed schema instead of building it for every degradationfails becauseJSONSchemer.schemais called for every degradation. -
To reproduce the measurements, download these scripts into
tmp/codequality-repro/(keepin_context/as a subfolder), with the head and base reports from the pipelines of !257526 (merged) inhead/andbase/:- bench.rb and variants.rb: isolated benchmark
- seed.rb: seeds a GDK merge request whose pipelines carry the reports as artifacts
- measure_direct.rb and measure_sidekiq.rb: in-context measurements
Then run each script on
masterand on this branch:LABEL=before VARIANT=baseline bin/rails runner tmp/codequality-repro/bench.rb bin/rails runner tmp/codequality-repro/in_context/seed.rb LABEL=before bin/rails runner tmp/codequality-repro/in_context/measure_direct.rb LABEL=before bin/rails runner tmp/codequality-repro/in_context/measure_sidekiq.rb
Out of scope
These came up during profiling and need separate decisions:
CodequalityReportsComparer#existing_errorsis always empty, becauseweb_urldiffers between head and base.- Degradations with the same fingerprint overwrite each other in
add_degradation. - Each degradation is still validated twice. With the schema cached, that is about 0.3s of the remaining 1.5s.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
