Implement usage metrics for ModSecurity Web Application Firewall

What does this MR do?

Adds usage stats to UsageData to measure counts of deployed applications utilizing the ModSecurity Web Application Firewall in 2 configurations:

• blocking - to block traffic violating default firewall rules
• disabled - to disable any processing of firewall rules (including logging)

Since we must query against 2 separate tables that are encrypted-at-rest we must decrypt and compare the values in memory to measure properly.

This implementation excludes variables set at the group level, this is a known limitation.

One of several MRs to address #32358 (closed)

This MR is a simplification of the previous !19374 (closed), with some minor code modifications but primarily the removal of the FeatureFlag check (in favor of !20194 (merged)) and addition of partial indices to improve query performance against the two variable tables

Queries

ci_pipeline_variables - https://explain.depesz.com/s/jImW

SELECT DISTINCT ON (deployments.environment_id) ci_pipeline_variables.* FROM "ci_pipeline_variables" INNER JOIN "ci_pipelines" ON "ci_pipelines"."id" = "ci_pipeline_variables"."pipeline_id" INNER JOIN "ci_builds" ON "ci_builds"."commit_id" = "ci_pipelines"."id" AND "ci_builds"."type" IN ('Ci::Build') INNER JOIN "deployments" ON "deployments"."deployable_id" = "ci_builds"."id" AND "deployments"."deployable_type" = 'CommitStatus' INNER JOIN "environments" ON "environments"."id" = "deployments"."environment_id" INNER JOIN "deployments" "last_visible_deployments_environments" ON "last_visible_deployments_environments"."environment_id" = "environments"."id" AND "last_visible_deployments_environments"."status" IN (1, 2, 3, 4) WHERE ("environments"."state" IN ('available')) AND "deployments"."status" = 2 AND "ci_pipeline_variables"."key" = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE' ORDER BY deployments.environment_id, deployments.id DESC

ci_variables - https://explain.depesz.com/s/BLMy

SELECT DISTINCT ON (deployments.environment_id) ci_variables.* FROM "ci_variables" INNER JOIN "projects" ON "projects"."id" = "ci_variables"."project_id" INNER JOIN "environments" ON "environments"."project_id" = "projects"."id" INNER JOIN "deployments" ON "deployments"."environment_id" = "environments"."id" AND "deployments"."status" IN (1, 2, 3, 4) WHERE ("environments"."state" IN ('available')) AND "deployments"."status" = 2 AND "ci_variables"."key" = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE' AND "environments"."id" NOT IN (SELECT DISTINCT ON (deployments.environment_id) deployments.environment_id FROM "ci_pipeline_variables" INNER JOIN "ci_pipelines" ON "ci_pipelines"."id" = "ci_pipeline_variables"."pipeline_id" INNER JOIN "ci_builds" ON "ci_builds"."commit_id" = "ci_pipelines"."id" AND "ci_builds"."type" IN ('Ci::Build') INNER JOIN "deployments" ON "deployments"."deployable_id" = "ci_builds"."id" AND "deployments"."deployable_type" = 'CommitStatus' INNER JOIN "environments" ON "environments"."id" = "deployments"."environment_id" INNER JOIN "deployments" "last_visible_deployments_environments" ON "last_visible_deployments_environments"."environment_id" = "environments"."id" AND "last_visible_deployments_environments"."status" IN (1, 2, 3, 4) WHERE ("environments"."state" IN ('available')) AND "deployments"."status" = 2 AND "ci_pipeline_variables"."key" = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE' ORDER BY deployments.environment_id, deployments.id DESC)```

Migration log

Migrating to AddIndexToModSecCiVariables (20191115001123)
   (0.2ms)  SET statement_timeout TO 0
  ↳ lib/gitlab/database/migration_helpers.rb:239
   (2.5ms)  CREATE  INDEX CONCURRENTLY "index_ci_variables_on_project_id" ON "ci_variables"  ("project_id") WHERE key = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE'
  ↳ lib/gitlab/database/migration_helpers.rb:93
   (0.2ms)  RESET ALL
  ↳ lib/gitlab/database/migration_helpers.rb:243
   (0.2ms)  BEGIN
  ↳ /Users/theoretick/.asdf/installs/ruby/2.6.3/bin/rake:23
  ActiveRecord::SchemaMigration Create (0.3ms)  INSERT INTO "schema_migrations" ("version") VALUES ($1) RETURNING "version"  [["version", "20191115001123"]]
  ↳ /Users/theoretick/.asdf/installs/ruby/2.6.3/bin/rake:23
   (0.6ms)  COMMIT
  ↳ /Users/theoretick/.asdf/installs/ruby/2.6.3/bin/rake:23
Migrating to AddIndexToModSecCiPipelineVariables (20191115001843)
   (0.1ms)  SET statement_timeout TO 0
  ↳ lib/gitlab/database/migration_helpers.rb:239
   (3.1ms)  CREATE  INDEX CONCURRENTLY "index_ci_pipeline_variables_on_pipeline_id" ON "ci_pipeline_variables"  ("pipeline_id") WHERE key = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE'
  ↳ lib/gitlab/database/migration_helpers.rb:93
   (0.1ms)  RESET ALL
  ↳ lib/gitlab/database/migration_helpers.rb:243
   (0.1ms)  BEGIN
  ↳ /Users/theoretick/.asdf/installs/ruby/2.6.3/bin/rake:23
  ActiveRecord::SchemaMigration Create (0.4ms)  INSERT INTO "schema_migrations" ("version") VALUES ($1) RETURNING "version"  [["version", "20191115001843"]]
  ↳ /Users/theoretick/.asdf/installs/ruby/2.6.3/bin/rake:23
   (0.5ms)  COMMIT

Database checklist

• Conforms to the database guides

When adding migrations:

• Updated db/schema.rb
• Added a down method so the migration can be reverted
• Added the output of the migration(s) to the MR body
• [-] Added tests for the migration in spec/migrations if necessary (e.g. when migrating data)
• Added rollback procedure. Include either a rollback procedure or description how to rollback changes

When adding or modifying queries to improve performance:

• Included data that shows the performance improvement, preferably in the form of a benchmark
• Included the output of EXPLAIN (ANALYZE, BUFFERS) of the relevant queries

When adding foreign keys to existing tables:

• Included a migration to remove orphaned rows in the source table before adding the foreign key
• Removed any instances of dependent: ... that may no longer be necessary

When adding tables:

• Ordered columns based on the Ordering Table Columns guidelines
• Added foreign keys to any columns pointing to data in other tables
• Added indexes for fields that are used in statements such as WHERE, ORDER BY, GROUP BY, and JOINs

When removing columns, tables, indexes or other structures:

• Removed these in a post-deployment migration
• Made sure the application no longer uses (or ignores) these structures