Implement usage metrics for ModSecurity Web Application Firewall
What does this MR do?
- Adds feature enablement to usage data since feature is disabled by default
- Adds usage stats to
UsageDatato measure counts of deployed applications utilizing the ModSecurity Web Application Firewall in 2 configurations:
-
blocking- to block traffic violating default firewall rules -
disabled- to disable any processing of firewall rules (including logging)
Since we must query against 2 separate tables that are encrypted-at-rest we must decrypt and compare the values in memory to measure properly... I don't love this solution, but not sure if there's something better.
Query Plans
::Ci::PipelineVariable
Unique (cost=678177.87..678181.81 rows=197 width=166) (actual time=4287.752..4287.752 rows=0 loops=1)
Buffers: shared hit=586 read=288357
I/O Timings: read=2563.568
-> Sort (cost=678177.87..678178.36 rows=197 width=166) (actual time=4287.739..4287.739 rows=0 loops=1)
Sort Key: ci_pipeline_variables.id, ci_pipeline_variables.value, ci_pipeline_variables.encrypted_value, ci_pipeline_variables.encrypted_value_salt, ci_pipeline_variables.encrypted_value_iv, ci_pipeline_variables.pipeline_id, ci_pipeline_variables.variable_type
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=586 read=288357
I/O Timings: read=2563.568
-> Nested Loop (cost=2.69..678170.36 rows=197 width=166) (actual time=4287.640..4287.640 rows=0 loops=1)
Buffers: shared hit=577 read=288357
I/O Timings: read=2563.568
-> Nested Loop (cost=2.27..677995.30 rows=295 width=170) (actual time=4287.626..4287.626 rows=0 loops=1)
Buffers: shared hit=577 read=288357
I/O Timings: read=2563.568
-> Nested Loop (cost=1.70..675951.04 rows=2695 width=170) (actual time=4287.613..4287.613 rows=0 loops=1)
Buffers: shared hit=577 read=288357
I/O Timings: read=2563.568
-> Nested Loop (cost=1.13..668498.11 rows=761 width=170) (actual time=4287.586..4287.586 rows=0 loops=1)
Buffers: shared hit=577 read=288357
I/O Timings: read=2563.568
-> Index Scan using index_ci_pipeline_variables_on_pipeline_id_and_key on ci_pipeline_variables (cost=0.56..666873.81 rows=761 width=166) (actual time=4287.585..4287.585 rows=0 loops=1)
Index Cond: ((key)::text = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE'::text)
Buffers: shared hit=577 read=288357
I/O Timings: read=2563.568
-> Index Only Scan using ci_pipelines_pkey on ci_pipelines (cost=0.57..2.12 rows=1 width=4) (never executed)
Index Cond: (id = ci_pipeline_variables.pipeline_id)
Heap Fetches: 0
-> Index Scan using index_ci_builds_on_commit_id_and_status_and_type on ci_builds (cost=0.57..9.02 rows=77 width=8) (never executed)
Index Cond: ((commit_id = ci_pipelines.id) AND ((type)::text = 'Ci::Build'::text))
-> Index Scan using index_deployments_on_deployable_type_and_deployable_id on deployments (cost=0.56..0.75 rows=1 width=8) (never executed)
Index Cond: (((deployable_type)::text = 'CommitStatus'::text) AND (deployable_id = ci_builds.id))
-> Index Scan using environments_pkey on environments (cost=0.42..0.58 rows=1 width=4) (never executed)
Index Cond: (id = deployments.environment_id)
Filter: ((state)::text = 'available'::text)
Planning time: 20.024 ms
Execution time: 4288.364 ms
::Ci::Variable
Unique (cost=701297.16..701297.19 rows=1 width=251) (actual time=3025.471..3025.471 rows=0 loops=1)
Buffers: shared hit=452 read=10066
I/O Timings: read=2947.690
-> Sort (cost=701297.16..701297.17 rows=1 width=251) (actual time=3025.470..3025.470 rows=0 loops=1)
Sort Key: ci_variables.id, ci_variables.value, ci_variables.encrypted_value, ci_variables.encrypted_value_salt, ci_variables.encrypted_value_iv, ci_variables.project_id, ci_variables.protected, ci_variables.environment_scope, ci_variables.masked, ci_variables.variable_type
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=452 read=10066
I/O Timings: read=2947.690
-> Nested Loop (cost=678635.52..701297.15 rows=1 width=251) (actual time=3025.390..3025.390 rows=0 loops=1)
Buffers: shared hit=447 read=10066
I/O Timings: read=2947.690
-> Nested Loop (cost=678635.10..701295.94 rows=2 width=255) (actual time=3025.390..3025.390 rows=0 loops=1)
Buffers: shared hit=447 read=10066
I/O Timings: read=2947.690
-> Index Scan using index_ci_variables_on_project_id_and_key_and_environment_scope on ci_variables (cost=678634.66..701290.52 rows=2 width=251) (actual time=3025.388..3025.388 rows=0 loops=1)
Index Cond: ((key)::text = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE'::text)
Filter: (NOT (hashed SubPlan 1))
Buffers: shared hit=447 read=10066
I/O Timings: read=2947.690
SubPlan 1
-> Unique (cost=678632.76..678633.75 rows=197 width=4) (never executed)
-> Sort (cost=678632.76..678633.25 rows=197 width=4) (never executed)
Sort Key: ci_pipelines.project_id
-> Nested Loop (cost=2.69..678625.25 rows=197 width=4) (never executed)
-> Nested Loop (cost=2.27..678450.20 rows=295 width=8) (never executed)
-> Nested Loop (cost=1.70..676405.94 rows=2695 width=8) (never executed)
-> Nested Loop (cost=1.13..668952.99 rows=761 width=12) (never executed)
-> Index Only Scan using index_ci_pipeline_variables_on_pipeline_id_and_key on ci_pipeline_variables (cost=0.56..666217.19 rows=761 width=4) (never executed)
Index Cond: (key = 'AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE'::text)
Heap Fetches: 0
-> Index Scan using ci_pipelines_pkey on ci_pipelines (cost=0.57..3.58 rows=1 width=8) (never executed)
Index Cond: (id = ci_pipeline_variables.pipeline_id)
-> Index Scan using index_ci_builds_on_commit_id_and_status_and_type on ci_builds (cost=0.57..9.02 rows=77 width=8) (never executed)
Index Cond: ((commit_id = ci_pipelines.id) AND ((type)::text = 'Ci::Build'::text))
-> Index Scan using index_deployments_on_deployable_type_and_deployable_id on deployments (cost=0.56..0.75 rows=1 width=8) (never executed)
Index Cond: (((deployable_type)::text = 'CommitStatus'::text) AND (deployable_id = ci_builds.id))
-> Index Scan using environments_pkey on environments environments_1 (cost=0.42..0.58 rows=1 width=4) (never executed)
Index Cond: (id = deployments.environment_id)
Filter: ((state)::text = 'available'::text)
-> Index Only Scan using projects_pkey on projects (cost=0.43..2.70 rows=1 width=4) (never executed)
Index Cond: (id = ci_variables.project_id)
Heap Fetches: 0
-> Index Only Scan using index_environments_on_project_id_and_state on environments (cost=0.42..0.55 rows=6 width=4) (never executed)
Index Cond: ((project_id = projects.id) AND (state = 'available'::text))
Heap Fetches: 0
Planning time: 23.118 ms
Execution time: 3025.747 msScreenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation created/updated or follow-up review issue created
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
Counting records in memory isn't great, but there's no other way to query this data otherwise unless we tie into the Deploy job with some kind of increment counter, but AFAIK we do not have such capability.
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Closes #32358 (closed)
Edited by Lucas Charles