Apply Postgres HLL distributed batch counter to Usage Ping secure pipelines metrics
What does this MR do?
It applies new experimental PostgresHllBatchDistinctCounter
added at !45673 (merged) to failing on gitlab.com secure pipeline metrics #267969 (comment 442049480)
Due to its experimental nature change is hidden behind the feature flag, also to avoid unintentionally breaking whole usage ping generation process, experimental usage of PostgresHllBatchDistinctCounter
silences all errors, logs them in sentry, and return -2
fallback value as indicator that something went wrong
Queries
- MIN query: https://explain.depesz.com/s/mvt6
- MAX query: https://explain.depesz.com/s/pv81
Different types of security scan types:
- https://explain.depesz.com/s/cfsn
- https://explain.depesz.com/s/i2No (reused warmed cached by 1st scan)
- https://explain.depesz.com/s/HReN (reused warmed cached by 1st scan)
- https://explain.depesz.com/s/dr4b (reused warmed cached by 1st scan)
- https://explain.depesz.com/s/Ei42 (reused warmed cached by 1st scan)
- https://explain.depesz.com/s/JHj2 Interesting case, planner selected different index scan strategy, execution time still low: 40ms
- https://explain.depesz.com/s/uIR1
Screenshots (strongly suggested)
Values from staging
Metric | Exact Value | Estimated Value | Error rate % |
---|---|---|---|
sast_pipeline | 742 | 732.9203393437964 | 1.223673942884588 |
dependency_scanning_pipeline | 776 | 771.3714194513371 | 0.5964665655493349 |
container_scanning_pipeline | 127 | 128.59755525713067 | 1.2579175252997317 |
dast_pipeline | 118 | 122.02547040891618 | 3.4114156007764223 |
secret_detection_pipeline | 64 | 67.44683389035757 | 5.385677953683697 |
coverage_fuzzing_pipeline | 0 | 0 | 0 |
api_fuzzing_pipeline | 0 | 0 | 0 |
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry change is behind feature flag
- [-] Documentation (if required) change is behind feature flag
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Edited by Mikołaj Wawrzyniak