Skip to content

Disable batch counting for "pipelines that have security jobs" metrics

What does this MR do?

Disable batch counting for "pipelines that have security jobs" metrics temporarily as it may challenge the large instances

The queries will fail on gitlab.com scale. But with batch turned on - it has performance risks for the general usage ping calculation.

  • Note that we have a cop to ban such counts "DistinctCountByLargeForeignKey" but it's disabled explicitly for the current ones

  • See monitoring gitlab.com usage ping issue #237851 (comment 413307197) - The counters fail since inception on gitlab.com

  • Counting over distinct pipeline id range would be a challenge. The reason is that, ci_pipelines table has 178 million rows currently. In contrast the users table has ~7 million rows

  • See issue #250269 (closed) and the MR which introduced those counters !37809 (merged)

Query Plans

SELECT COUNT(DISTINCT "ci_builds"."commit_id") FROM "ci_builds" INNER JOIN "security_scans" ON "security_scans"."build_id" = "ci_builds"."id" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."status" = 'success' AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND (security_scans.scan_type = 1) AND "ci_builds"."created_at" BETWEEN '2020-08-20 05:51:06.661009' AND '2020-09-17 05:51:06.661080'

SELECT COUNT(DISTINCT "ci_builds"."commit_id") FROM "ci_builds" INNER JOIN "security_scans" ON "security_scans"."build_id" = "ci_builds"."id" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."status" = 'success' AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND (security_scans.scan_type = 2) AND "ci_builds"."created_at" BETWEEN '2020-08-20 05:51:06.661009' AND '2020-09-17 05:51:06.661080'

SELECT COUNT(DISTINCT "ci_builds"."commit_id") FROM "ci_builds" INNER JOIN "security_scans" ON "security_scans"."build_id" = "ci_builds"."id" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."status" = 'success' AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND (security_scans.scan_type = 3) AND "ci_builds"."created_at" BETWEEN '2020-08-20 05:51:06.661009' AND '2020-09-17 05:51:06.661080'

SELECT COUNT(DISTINCT "ci_builds"."commit_id") FROM "ci_builds" INNER JOIN "security_scans" ON "security_scans"."build_id" = "ci_builds"."id" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."status" = 'success' AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND (security_scans.scan_type = 4) AND "ci_builds"."created_at" BETWEEN '2020-08-20 05:51:06.661009' AND '2020-09-17 05:51:06.661080'

SELECT COUNT(DISTINCT "ci_builds"."commit_id") FROM "ci_builds" INNER JOIN "security_scans" ON "security_scans"."build_id" = "ci_builds"."id" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."status" = 'success' AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND (security_scans.scan_type = 5) AND "ci_builds"."created_at" BETWEEN '2020-08-20 05:51:06.661009' AND '2020-09-17 05:51:06.661080'

SELECT COUNT(DISTINCT "ci_builds"."commit_id") FROM "ci_builds" INNER JOIN "security_scans" ON "security_scans"."build_id" = "ci_builds"."id" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."status" = 'success' AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND (security_scans.scan_type = 6) AND "ci_builds"."created_at" BETWEEN '2020-08-20 05:51:06.661009' AND '2020-09-17 05:51:06.661080'

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Alper Akgun

Merge request reports