Add CI-based analyzer observability metrics documentation

Product direction can be found on the [Sec Section Product Direction](https://about.gitlab.com/direction/security/) handbook page.

## Sec Section Resources

The following resources provide guidance for common development patterns across Sec Section teams:

### Observability

- [Tutorial: Add observability metrics to a CI-based analyzer](/handbook/engineering/development/sec/secure/analyzer-observability-metrics/) -

  Step-by-step guide for implementing the decentralized events pattern in security analyzers.

- [Secret Detection metrics](/handbook/engineering/development/sec/secure/secret-detection/metrics/) -

  Guide for adding metrics to the Secret Detection analyzer and GitLab monolith.

## Project Setup

Keeping our projects organized is very important for productivity and maintainability.

---

title: "Tutorial: Add observability metrics to a CI-based analyzer"

---

Observability metrics help you understand how CI-based security analyzers perform in production.

Metrics like scan duration, exit codes, and findings count, flow through the security report

artifact to GitLab's internal event tracking system.

For an implementation example, see

[Secret Detection metrics](/handbook/engineering/development/sec/secure/secret-detection/metrics/#ci-based-analyzer-observability-metrics).

## Architecture

```mermaid

flowchart LR

    A[CI Analyzer] --> B[Security Report]

    B --> C[Post-analyzer]

    C --> D[Internal Event Tracking]

    D --> E[Snowflake]

```

1. The CI analyzer collects metrics during the scan and writes them to the security report artifact.

1. The post-analyzer processes the report and extracts observability events.

1. Allowed events are sent to the internal event tracking system.

1. Events are stored in Snowflake for analysis and dashboards.

## Design considerations

The observability system uses a decentralized events pattern. Each analyzer defines its own

event structure rather than using a shared registry.

This design provides:

- **Independent development**: Add or modify metrics without coordinating releases

  across repositories.

- **Version tolerance**: Older post-analyzers skip unknown events gracefully instead of failing.

- **Faster iteration**: Test new metrics locally without infrastructure updates.

- **Decentralized ownership**: Each analyzer team owns their event definitions.

The trade-offs are:

- No single location lists all possible events. Use the monolith event definitions in

  [`config/events/`](https://gitlab.com/gitlab-org/gitlab/-/tree/v18.6.4-ee/config/events) or

  [`ee/config/events/`](https://gitlab.com/gitlab-org/gitlab/-/tree/v18.6.4-ee/ee/config/events)

  as the source of truth.

- New events aren't processed until the post-analyzer and backend are updated.

## Prerequisites

The observability feature requires:

- [Report](https://gitlab.com/gitlab-org/security-products/analyzers/report) package v7 or later

- [Command](https://gitlab.com/gitlab-org/security-products/analyzers/command) package v5 or later

## Event design guidance

- Create multiple events rather than using many extra properties. JSON columns with extra

  properties are slower to query.

- To join related events, include a UUID in a column value (for example, `property`).

- Before adding a variable field, check with maintainers of the

  [integration-test](https://gitlab.com/gitlab-org/security-products/tests/integration-test)

  project to ensure it will be handled correctly.

## Register the internal event

Events require a corresponding definition in the GitLab monolith and must be added to the

backend allow list. Only allowed events are processed.

1. Create event definition in `config/events/` or `ee/config/events/`.

1. Update the allow list in

   [ProcessScanEventsService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/security/process_scan_events_service.rb#L9).

1. Add an rspec test using actual report output from the analyzer.

For more information, see

[Quick start for internal event tracking](https://docs.gitlab.com/ee/development/internal_analytics/internal_event_instrumentation/quick_start.html).

## Validation

After deployment:

1. Query Snowflake to verify events are collected correctly.

1. Check Sentry for exceptions during report processing.

1. Request dashboard creation from the Analytics Instrumentation team.

## Overview

This page documents the process a member of the Secure: Secret Detection team should use to add metrics to capture product insights for the features we develop.

This page documents the process a member of the Secure: Secret Detection team should use to add

metrics to capture product insights for the features we develop.

You should this guide to help you understand the 2 different types of metrics we

You should use this guide to help you understand the different types of metrics we

can use, when to use each one, and give you a jumpstart in implementing them.

### Metrics workflow

(formerly Service Ping) depending on the situation:

- Internal Tracking Events are good for capturing _events_ that occur but aren't

stored in the database. E.g., a user clicks a specific button.

stored in the database. For example, a user clicks a specific button.

- Database metrics are useful when the data you want _is_ stored in the database

in some way that can be extracted with the right query. E.g., the nubmer of

in some way that can be extracted with the right query. For example, the number of

projects that have a setting enabled.

The Analytics Instrumentation team has great documentation

#### Process for adding

You should follow the process Analytics Instrumentation has defined in the [quick start guide](https://docs.gitlab.com/ee/development/internal_analytics/internal_event_instrumentation/quick_start.html#quick-start-for-internal-event-tracking).

linked docs.

TL;DR Run the `ruby scripts/internal_events/cli.rb` CLI tool and follow the

Run the `ruby scripts/internal_events/cli.rb` CLI tool and follow the

prompts. The event definition is necessary for the code to know what

events to output and the metric definition is what Snowflake and Tableau will

make available given the events.

    ...

    subject

end

```

Unlike in the implementation file, when using the shared example in the specs,

rails generate gitlab:usage_metric CountIssues --type database --operation distinct_count

        create lib/gitlab/usage/metrics/instrumentations/count_issues_metric.rb

        create spec/lib/gitlab/usage/metrics/instrumentations/count_issues_metric_spec.rb

```

The simplest way to implement the metric is to call the class-level `#operation` and `#relation` methods.

`time_frame` should match the value in the dictionary for the metric that was

defined, i.e., `7d`, `28d`, or `all`.

## CI-based analyzer observability metrics

This section covers adding observability metrics to the

[Secret Detection analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/secrets).

For architecture and design considerations, see

[Tutorial: Add observability metrics to a CI-based analyzer](/handbook/engineering/development/sec/secure/analyzer-observability-metrics/).

For implementation details on adding new events, see the

[analyzer README](https://gitlab.com/gitlab-org/security-products/analyzers/secrets#adding-observability-events).

### Register the event

Events require a corresponding definition in the GitLab monolith and must be added to the

backend allow list. Only allowed events are processed.

1. Create event definition in

   [`config/events/`](https://gitlab.com/gitlab-org/gitlab/-/tree/v18.6.4-ee/config/events) or

   [`ee/config/events/`](https://gitlab.com/gitlab-org/gitlab/-/tree/v18.6.4-ee/ee/config/events).

1. Add the new event name to

   [Security::ProcessScanEventsService::EVENT_NAME_ALLOW_LIST](https://gitlab.com/gitlab-org/gitlab/-/blob/v18.6.4-ee/ee/app/services/security/process_scan_events_service.rb#L9-46).

1. Add an rspec test using actual report output from the analyzer.

For more information, see

[Quick start for internal event tracking](https://docs.gitlab.com/ee/development/internal_analytics/internal_event_instrumentation/quick_start.html).

### Validation

After deployment:

1. Query Snowflake to verify events are collected correctly.

1. Check Sentry for exceptions during report processing.

1. Request dashboard creation from the Analytics Instrumentation team.

### Current implementation

The Secret Detection analyzer emits the `collect_secret_detection_scan_metrics_from_pipeline` event.

| Field | Description |

|-------|-------------|

| `event` | Event name. Always `collect_secret_detection_scan_metrics_from_pipeline`. |

| `time_s` | Scan duration in seconds. |

| `exit_code` | Gitleaks exit code. `0` for success, non-zero for errors or findings. |

| `git_strategy` | Git fetch strategy: `FetchRange`, `FetchAll`, or `NoFetch`. |

| `commit_count` | Number of commits scanned. Zero for directory scans. |

| `bytes_scanned` | Approximate bytes scanned by Gitleaks. |

| `leaks_found` | Number of secrets detected. |

| `repo_size_kb` | Repository size in kilobytes. |

| `error_message` | Error message if the scan failed. |

### Related merge requests

- [Add observability metrics to Secret Detection analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/merge_requests/428)

## Viewing and analyzing

### Verifying creation and deployment