Skip to content

Update batch size when calculating secure build metrics

Alan (Maciej) Paruszewski requested to merge 344264-update-batch-size-for-secure-jobs-metric into master

What does this MR do and why?

During the review of my last MR @marc_shaw discovered the reason that we are unable to calculate metrics for jobs: !77991 (comment 818027397). We need to increase the batch_size to reduce the number of MAX_LOOPS (10_000) (verified here).

This MR addresses that by extending lib/gitlab/usage/metrics/instrumentations/database_metric.rb with ability to specify additional options passed to metric operation and by setting custom batch_size value for ee/lib/gitlab/usage/metrics/instrumentations/count_ci_builds_metric.rb.

Based on the ID from the latest Job (https://gitlab.com/gitlab-org/gitlab/-/jobs) - 2097825284 and by changing batch_size to 1_000_000, we would have 2097825284 / 1000000 = ~2100 loops which will work as expected.

Queries

  1. Batch 1_000_000:
explain SELECT COUNT("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning' AND "ci_builds"."id" >= 2096667775 AND "ci_builds"."id" < 2097667775;
Time: 32.193 ms
  - planning: 31.615 ms
  - execution: 0.578 ms
    - I/O read: 0.377 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 3 (~24.00 KiB) from the buffer pool
  - reads: 4 (~32.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/8642/commands/30693

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #344264 (closed)

Edited by Alan (Maciej) Paruszewski

Merge request reports