Add metrics for Threat Insights Group

Why are we doing this work

In Govern SMAU Metric, we want to add tracking of govern-related activities. The metric should report only unique users and should use user ID as a tracking value, so that events tracked from frontend via /increment_unique_users don't create duplicates. There is a PoC which explored the possible implementation.

We want to generate metrics for each page, aggregate metrics for each of the 5 categories, aggregate metrics for each of the 3 groups and one aggregate metric for the entire stage.

In this issue, we want to add tracking for activities related to the Threat Insights Group.

Relevant links

• Govern SMAU Metric
• PoC

Implementation plan

• backend Add metrics to the list of known_events:

- name: users_visiting_security_vulnerability_report
  aggregation: weekly
- name: users_visiting_security_dashboard
aggregation: weekly
- name: users_visiting_security_vulnerabilities
  aggregation: weekly
- name: users_visiting_pipeline_security
aggregation: weekly
- name: users_visiting_security_configuration_threat_management
aggregation: weekly
- name: users_visiting_dependencies
  aggregation: weekly
- name: users_visiting_licenses
  aggregation: weekly
- name: users_visiting_vulnerability_management_pages
aggregation: weekly
- name: users_visiting_dependency_management_pages
aggregation: weekly
- name: users_visiting_threat_insights_pages
aggregation: weekly

• backend Generate usage metric definitions for the aggregated vulnerability management category metric:

bin/rails generate gitlab:usage_metric_definition:redis_hll govern users_visiting_vulnerability_management_pages --ee

Use AggregatedMetric as instrumentation class and include the events to aggregate:

instrumentation_class: AggregatedMetric
options:
  aggregate:
    operator: OR
    attribute: user_id
  events:
    - users_visiting_security_vulnerability_report
    - users_visiting_security_dashboard
    - users_visiting_security_vulnerabilities
    - users_visiting_pipeline_security
    - users_visiting_security_configuration_threat_management
    - i_code_review_merge_request_widget_security_reports_expand

• backend Generate usage metric definitions for the aggregated dependency management category metric:

bin/rails generate gitlab:usage_metric_definition:redis_hll govern users_visiting_dependency_management_pages --ee

Use AggregatedMetric as instrumentation class and include the events to aggregate:

instrumentation_class: AggregatedMetric
options:
  aggregate:
    operator: OR
    attribute: user_id
  events:
    - users_visiting_dependencies
    - users_visiting_licenses

• backend Generate usage metric definition for the aggregated group metric:

bin/rails generate gitlab:usage_metric_definition:redis_hll govern users_visiting_threat_insights_pages --ee

Use AggregatedMetric as instrumentation class and include the events to aggregate:

instrumentation_class: AggregatedMetric
options:
  aggregate:
    operator: OR
    attribute: user_id
  events:
    - users_visiting_security_vulnerability_report
    - users_visiting_security_dashboard
    - users_visiting_security_vulnerabilities
    - users_visiting_pipeline_security
    - users_visiting_security_configuration_threat_management
    - users_visiting_dependencies
    - users_visiting_licenses
    - i_code_review_merge_request_widget_security_reports_expand

• backend Add the events above also to the aggregated SMAU metric:
  • users_visiting_govern_pages_monthly
  • users_visiting_govern_pages_weekly
• backend Add tracking of project-level actions in the controllers using GovernUsageProjectTracking:
  • index in ee/app/controllers/projects/security/dashboard_controller.rb
  • security in ee/app/controllers/ee/projects/pipelines_controller.rb
  • %i[index show] in ee/app/controllers/projects/security/vulnerabilities_controller.rb
  • index in ee/app/controllers/projects/security/vulnerability_report_controller.rb
  • index in ee/app/controllers/projects/security/dashboard_controller.rb
  • index in ee/app/controllers/projects/dependencies_controller.rb
  • index in ee/app/controllers/projects/licenses_controller.rb
• backend Add tracking of group-level actions in the controllers using GovernUsageGroupTracking:
  • show in ee/app/controllers/groups/security/dashboard_controller.rb
  • index in ee/app/controllers/groups/dependencies_controller.rb
• frontend Add tracking of users_visiting_security_configuration_threat_management when threat management tab is selected (app/assets/javascripts/security_configuration/components/app.vue)
• frontend Add tracking of users_visiting_pipeline_security when security tab is selected (ee/app/assets/javascripts/pipelines/components/pipeline_tabs.vue)

Verification steps

1. Visit one of the pages or perform any of the actions covered by the new events
2. In the rails console, check for a non-zero counter value for the metric:

   Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: '<event-name>', start_date: Date.current.beginning_of_week, end_date: Date.current.next_week)

3. You run the service ping worker with GitlabServicePingWorker.new.perform('triggered_from_cron' => false)
4. Check for the weekly and monthly aggregated metric:

   RawUsageData.last.payload['redis_hll_counters']['govern']

Sisense

      SELECT
        ping_created_at,
        metrics_path,
        metric_value,
        has_timed_out --if metric timed out, the value will be set to 0
      FROM common.fct_ping_instance_metric_rolling_6_months --model limited to last 6 months for performance
      WHERE dim_instance_id = 'ea8bf810-1d6f-4a6a-b4fd-93e8cbd8b57f' --SaaS instance, both production and staging installations
      AND metrics_path = 'redis_hll_counters.govern.users_visiting_threat_insights_pages_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_vulnerability_management_pages_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_dependency_management_pages_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_security_vulnerability_report_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_security_dashboard_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_security_vulnerabilities_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_pipeline_security_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_dependencies_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_licenses_monthly'
      OR metrics_path = 'redis_hll_counters.govern.users_visiting_security_configuration_threat_management_monthly'
      ORDER BY ping_created_at DESC
      LIMIT 10
      ;