Telemetry for secret detection

What does this MR do?

This is a follow-up from this MR. This MR adds telemetry and db index changes related to secret-detection. The original issues related to this MR are as follows:

• #204987 (closed)
• #215082 (closed)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

Performance Analysis

CREATE INDEX index_ci_builds_on_name_and_security_type_eq_ci_build_2 ON public.ci_builds USING btree (name, id) WHERE (((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('secret_detection'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text));

The query has been executed. Duration: 110.629 min

Query Type 1

pry(main)> Gitlab::UsageData.count(::Ci::Build.where(name: secure_type))
   (2.7ms)  SELECT MIN("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'
   (0.5ms)  SELECT MAX("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'
   (0.5ms)  SELECT COUNT("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."id" BETWEEN 0 AND 99999

SELECT MIN("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'

• before: https://explain.depesz.com/s/Ip9u
• after: https://explain.depesz.com/s/TJue

SELECT MAX("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'

• before: https://explain.depesz.com/s/ciH3
• after: https://explain.depesz.com/s/BSyD

SELECT COUNT("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."id" BETWEEN 0 AND 99999

• before: https://explain.depesz.com/s/HDEv
• after: https://explain.depesz.com/s/f3Mx

Query Type 2

[5] pry(main)> pry(main)> time_period = {}
NameError: undefined local variable or method `main' for main:Object
from (pry):5:in `<main>'
[6] pry(main)> time_period = {}
=> {}
[7] pry(main)> Gitlab::UsageData.distinct_count(::Ci::Build.where(name: secure_type).where(time_period), :user_id)
   (0.7ms)  SELECT MIN("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'
   (0.6ms)  SELECT MAX("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'
   (0.6ms)  SELECT COUNT(DISTINCT "ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."user_id" BETWEEN 0 AND 9999

SELECT MIN("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'

• before: https://explain.depesz.com/s/mi69
• after: https://explain.depesz.com/s/ALJY

SELECT MAX("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection'

• before: https://explain.depesz.com/s/dGAi
• after: https://explain.depesz.com/s/iaKX

SELECT COUNT(DISTINCT "ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."user_id" BETWEEN 0 AND 9999

• before: https://explain.depesz.com/s/L1yP
• after: https://explain.depesz.com/s/2G6O

Query Type 3

[8] pry(main)> time_period = { created_at: 28.days.ago..Time.current }
=> {:created_at=>Tue, 21 Apr 2020 21:40:24 UTC +00:00..Tue, 19 May 2020 21:40:24 UTC +00:00}
[9] pry(main)> Gitlab::UsageData.distinct_count(::Ci::Build.where(name: secure_type).where(time_period), :user_id)
   (0.2ms)  SELECT "features"."key" FROM "features"
   (0.7ms)  SELECT MIN("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."created_at" BETWEEN '2020-04-21 21:40:24.288642' AND '2020-05-19 21:40:24.288763'
   (0.6ms)  SELECT MAX("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."created_at" BETWEEN '2020-04-21 21:40:24.288642' AND '2020-05-19 21:40:24.288763'
   (0.3ms)  SELECT COUNT(DISTINCT "ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."created_at" BETWEEN '2020-04-21 21:40:24.288642' AND '2020-05-19 21:40:24.288763' AND "ci_builds"."user_id" BETWEEN 0 AND 9999

SELECT MIN("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."created_at" BETWEEN '2020-04-21 21:40:24.288642' AND '2020-05-19 21:40:24.288763'

• before: https://explain.depesz.com/s/qLrq
• after: https://explain.depesz.com/s/9jDwE

SELECT MAX("ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."created_at" BETWEEN '2020-04-21 21:40:24.288642' AND '2020-05-19 21:40:24.288763'

• before: https://explain.depesz.com/s/IBKR
• after: https://explain.depesz.com/s/jQY4

SELECT COUNT(DISTINCT "ci_builds"."user_id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'secret_detection' AND "ci_builds"."created_at" BETWEEN '2020-04-21 21:40:24.288642' AND '2020-05-19 21:40:24.288763' AND "ci_builds"."user_id" BETWEEN 0 AND 9999

• before: https://explain.depesz.com/s/jBBdx
• after: https://explain.depesz.com/s/IG1m

Migration Output

➜  gitlab git:(telemetry-for-secret-detection) ✗ VERBOSE=true bin/rails db:migrate:up VERSION=20200512195442 RAILS_ENV=development

== 20200512195442 UpdateIndexOnNameTypeEqCiBuildToCiBuilds: migrating =========
-- transaction_open?()
 -> 0.0000s
-- index_exists?(:ci_builds, [:name, :id], {:name=>"index_security_ci_builds_on_name_and_id", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text,\n                ('dast'::character varying)::text,\n                ('dependency_scanning'::character varying)::text,\n                ('license_management'::character varying)::text,\n                ('sast'::character varying)::text,\n                ('secret_detection'::character varying)::text,\n                ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
 -> 0.0086s
-- add_index(:ci_builds, [:name, :id], {:name=>"index_security_ci_builds_on_name_and_id", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text,\n                ('dast'::character varying)::text,\n                ('dependency_scanning'::character varying)::text,\n                ('license_management'::character varying)::text,\n                ('sast'::character varying)::text,\n                ('secret_detection'::character varying)::text,\n                ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
 -> 0.0040s
-- transaction_open?()
 -> 0.0000s
-- indexes(:ci_builds)
 -> 0.0064s
-- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>"index_ci_builds_on_name_and_security_type_eq_ci_build"})
 -> 0.0014s
== 20200512195442 UpdateIndexOnNameTypeEqCiBuildToCiBuilds: migrated (0.0209s)

➜  gitlab git:(telemetry-for-secret-detection) VERBOSE=true bin/rails db:migrate:down VERSION=20200512195442 RAILS_ENV=development

== 20200512195442 UpdateIndexOnNameTypeEqCiBuildToCiBuilds: reverting =========
-- transaction_open?()
 -> 0.0000s
-- index_exists?(:ci_builds, [:name, :id], {:name=>"index_ci_builds_on_name_and_security_type_eq_ci_build", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text,\n                ('dast'::character varying)::text,\n                ('dependency_scanning'::character varying)::text,\n                ('license_management'::character varying)::text,\n                ('sast'::character varying)::text,\n                ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
 -> 0.0094s
-- add_index(:ci_builds, [:name, :id], {:name=>"index_ci_builds_on_name_and_security_type_eq_ci_build", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text,\n                ('dast'::character varying)::text,\n                ('dependency_scanning'::character varying)::text,\n                ('license_management'::character varying)::text,\n                ('sast'::character varying)::text,\n                ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
 -> 0.0034s
-- transaction_open?()
 -> 0.0000s
-- indexes(:ci_builds)
 -> 0.0058s
-- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>"index_security_ci_builds_on_name_and_id"})
 -> 0.0013s
== 20200512195442 UpdateIndexOnNameTypeEqCiBuildToCiBuilds: reverted (0.0205s)