Skip to content

Apply Postgres HLL distributed batch counter to Usage Ping secure pipelines metrics

What does this MR do?

It applies new experimental PostgresHllBatchDistinctCounter added at !45673 (merged) to failing on gitlab.com secure pipeline metrics #267969 (comment 442049480)

Due to its experimental nature change is hidden behind the feature flag, also to avoid unintentionally breaking whole usage ping generation process, experimental usage of PostgresHllBatchDistinctCounter silences all errors, logs them in sentry, and return -2 fallback value as indicator that something went wrong

Queries

Different types of security scan types:

  1. https://explain.depesz.com/s/cfsn
  2. https://explain.depesz.com/s/i2No (reused warmed cached by 1st scan)
  3. https://explain.depesz.com/s/HReN (reused warmed cached by 1st scan)
  4. https://explain.depesz.com/s/dr4b (reused warmed cached by 1st scan)
  5. https://explain.depesz.com/s/Ei42 (reused warmed cached by 1st scan)
  6. https://explain.depesz.com/s/JHj2 Interesting case, planner selected different index scan strategy, execution time still low: 40ms
  7. https://explain.depesz.com/s/uIR1

Screenshots (strongly suggested)

Values from staging

Metric Exact Value Estimated Value Error rate %
sast_pipeline 742 732.9203393437964 1.223673942884588
dependency_scanning_pipeline 776 771.3714194513371 0.5964665655493349
container_scanning_pipeline 127 128.59755525713067 1.2579175252997317
dast_pipeline 118 122.02547040891618 3.4114156007764223
secret_detection_pipeline 64 67.44683389035757 5.385677953683697
coverage_fuzzing_pipeline 0 0 0
api_fuzzing_pipeline 0 0 0

Does this MR meet the acceptance criteria?

Conformity

Edited by Mikołaj Wawrzyniak

Merge request reports