Collecting CI-based analyzer metrics
## Problem
Collecting metrics from CI-based analyzers is difficult because the CI-based analyzers don't have access the existing observability functionality in the GitLab monolith. This has created a situation in which the analyzer teams don't have the necessary data to make data driven decisions about ongoing development.
The ~"group::analytics instrumentation" is looking to pair with ~"devops::secure" teams to help with adding metrics. [Epic is here.](https://gitlab.com/gitlab-org/gitlab/-/issues/461819 "Pair with Secure stage on metric/event instrumentation") They could be a useful resource for this work.
## Adding Metrics
For instructions on adding metrics [to CI-based analyzers, see this sub-epic.](https://gitlab.com/groups/gitlab-org/-/epics/14555)
NOTE: This feature is still behind a FF which you can enable to test changes once merged. It's expected that `gitlab-org` will be enabled first week of April 2025.
## [FY26 Roadmap Item](https://docs.google.com/presentation/d/1ABoGLJkQZNs3Y92NELNrRvjsbo_PNEjGMyCRVz2sU2A/edit#slide=id.g3321f028e83_21_26)
| Roadmap Issue | Deliverable | Status |
| ------------- | ----------- | ------ |
| https://gitlab.com/gitlab-org/gitlab/-/issues/473109+s | ~Deliverable | :heavy_check_mark: |
| https://gitlab.com/gitlab-org/gitlab/-/issues/477257+s | ~Stretch | |
## Using the security report
A method for collecting metrics from CI-based analyzers that uses existing artifacts (the security report) to capture metrics which are then processed by the backend during security report processing and reported via the existing [internal event tracking.](https://docs.gitlab.com/ee/development/internal_analytics/internal_event_instrumentation/)
```mermaid
sequenceDiagram
box CI Pipeline
participant A as Analyzer
participant R as Runner
end
box Monolith
participant I as Ingestion
participant E as Events
end
A->>A: Create security report
A->>A: [New] Add metrics to report
A->>R: Report artifact
R->>I: Process report
loop [New] For each metric
I->>E: [New] Log metric
end
```
Assumptions:
1. Metric data will not be excessive enough to cause report size issues.
Adding a new metric:
1. [Define the event in the monolith](https://docs.gitlab.com/ee/development/internal_analytics/internal_event_instrumentation/event_definition_guide.html). Recommend working with ~"group::analytics instrumentation" when defining metrics to collect.
1. Add metric to the security report when the report is generated
1. Verify data collection using snowflake queries
1. Consider adding a service ping aggregation to get data from self managed instances
### Internal event tracking
Internal event tracking allows tracking events with 3 properties that map to columns in the data warehouse and any number of custom properties that end up in a JSON column. The ability to store any number of custom properties with events is [a new feature that is not yet documented (first users!).](https://gitlab.com/gitlab-org/gitlab/-/issues/462378)
Events can relate to other events using an identifier (like a scan ID). The 3 properties that map to columns are used for core filtering values as they are the most performant.
_Example from proposed schema changes:_
In this example two events are tracked that are related via the `property` property.
The format of the JSON entries in the security report are derived from the internal event tracking API signatures.
The first event `api_security_scan` occurs once per analyzer run and contains top level metrics that might be used in a tableau dashboard. the `property` property of the event is set to the scan uuid identifier. This will allow relating other events to this scan.
The second event `api_security_cwe_92`, are metrics about a SQL Injection security check performed by the scanner. This event also contains the scan uuid identifier in the `property` property allowing it to be related back to the `api_security_scan` event.
```json
{
"scan": {
"observability": {
"events": [
{
# Name of the event
"event": "api_security_scan",
# Following 3 properties are columns in the data warehouse
# they should be used for primary join/filtering
"property": "scan_uuid", # Allows joining across events
"label": "openapi|graphql_schema|postman|graphql|har",
"value": 60, # length of scan in minutes
# Custom properties stored in a JSON column in the data warehouse
"operations": 5
"version": "5.5.5",
"version_major": 5,
"operations": 10, # How many API operations we are scanning
"per-request-script": true,
"overrides-command": true,
"status": "completed|failed",
"content-type": "mixed|graphql|json|xml|soap",
},
{
# This event can be joined via the scan_uuid
"event": "api_security_cwe_92",
"property": "scan_uuid",
"label": "graphql",
"value": 5, # count of vulnerabilities
"parameters": 20, # number of parameters check ran on
},
],
}}
}
```
<!-- start-discoto-summary -->
## Auto-Summary :robot:
<details>
<summary>Discoto Usage</summary>
---
> **Points**
>
> Discussion points are declared by headings, list items, and single
> lines that start with the text (case-insensitive) `point:`. For
> example, the following are all valid points:
>
> * `#### POINT: This is a point`
> * `* point: This is a point`
> * `+ Point: This is a point`
> * `- pOINT: This is a point`
> * `point: This is a **point**`
>
> Note that any markdown used in the point text will also be propagated
> into the topic summaries.
>
> **Topics**
>
> Topics can be stand-alone and contained within an issuable (epic,
> issue, MR), or can be inline.
>
> Inline topics are defined by creating a new thread (discussion)
> where the first line of the first comment is a heading that starts
> with (case-insensitive) `topic:`. For example, the following are all
> valid topics:
>
> * `# Topic: Inline discussion topic 1`
> * `## TOPIC: **{+A Green, bolded topic+}**`
> * `### tOpIc: Another topic`
>
> **Quick Actions**
>
> | Action | Description |
> |-------------------------------|---------------------------------------------------------|
> | `/discuss sub-topic TITLE` | Create an issue for a sub-topic. Does not work in epics |
> | `/discuss link ISSUABLE-LINK` | Link an issuable as a child of this discussion |
>
---
</details>
Last updated by [this job](https://gitlab.com/gitlab-org/secure/pocs/discoto-runner/-/jobs/7990029201)
* **TOPIC** Define baseline metrics for measuring job success https://gitlab.com/groups/gitlab-org/-/issues/14266#note_2142052210
<!-- end-discoto-summary -->
<!-- start-discoto-topic-settings --><details>
<summary>Discoto Settings</summary>
<br/>
```yaml
---
summary:
max_items: -1
sort_by: created
sort_direction: ascending
```
See the [settings schema](https://gitlab.com/gitlab-org/secure/pocs/discoto#settings-schema) for details.
</details>
<!-- end-discoto-topic-settings -->
epic