## Background ### The Feature Issue [#384960](https://gitlab.com/gitlab-org/gitlab/-/work_items/384960) introduced the ability to detect and distinguish between different Git operations (clone vs pull/fetch vs push) in audit logs. This allows users to see not just "someone accessed the repository" but specifically "who did what action" (e.g., "Jane cloned the repo", "CI runner pulled the latest code"). ### What Happened The team implemented this feature behind a feature flag (`log_git_streaming_audit_events`) and attempted to roll it out globally in version 18.3. However, the rollout was reverted due to high volumes of 404/401 errors flooding the logs. ### Root Cause The issue stems from how GitLab identifies actors in audit events: **Current Behavior:** - When a regular user (e.g., Jane) runs `git clone`, GitLab logs: `"author": "jane"` - When a CI/CD runner or production server runs `git clone` using a deploy token, GitLab receives: `"author": "deploy-token-123"` - GitLab's `/shellhorse/git_audit_event` endpoint only recognizes username-based identifiers - When it receives a deploy token identifier, it returns 404/401 errors **Why This Matters:** Git operations don't only come from regular users. They also come from: - CI/CD runners (using deploy tokens) - Production servers (using deploy tokens) - Automated systems (using SSH certificates) - Keyless SSH operations All of these use token-based or certificate-based authentication instead of usernames. ### Current Status - Feature flag is disabled - Rollout is blocked by two issues: - [#562516](https://gitlab.com/gitlab-org/gitlab/-/work_items/562516): Fix authentication for HTTP-based Git operations (Workhorse) - [#822](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/822): Fix authentication for SSH-based Git operations (gitlab-shell) ### Implementation #### Phase 1: Spike(18.9) | Issue | Weight |Milestone |Repo | Done? | | ------ | ------ | ------ |------ |------ | | [Spike: Investigate Git Streaming Audit Events Error](https://gitlab.com/gitlab-org/gitlab/-/work_items/586358) | 3 | 18.9| gitlab & gitlab-shell | :white_check_mark: | #### Phase 2: HTTP fix | Issue | Weight |Milestone |Repo |Done?| | ------ | ------ | ------ |------ |------ | | [Add human actor guard to GitAuditEvent before re-enabling feature flag](https://gitlab.com/gitlab-org/gitlab/-/work_items/591285) |1 |18.10| gitlab |:white_check_mark: | | [Enable log_git_streaming_audit_events FF for human actors only ](https://gitlab.com/gitlab-org/gitlab/-/work_items/591572)|1| 18.10|gitlab| :white_check_mark: Enabled in prod | | [Introduce deploy-token-{id} identifier](https://gitlab.com/gitlab-org/gitlab/-/work_items/570235) | 2 | 18.10 | gitlab | :white_check_mark: [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/223145) merged| | [Fix GlId generation for DeployToken](https://gitlab.com/gitlab-org/gitlab/-/work_items/570234) | 2 | 18.10 |gitlab | :white_check_mark: [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/222599) merged| | [Migrate Audit system to use identifier](https://gitlab.com/gitlab-org/gitlab/-/work_items/570236) | 2 | 18.10 |gitlab | 1. [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/224363) in dev / 2. [A draft MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/224389)| #### Phase 3: SSH fix | Issue | Weight | Milestone |Repo |Done?| | ------ | ------ | ------| ------ |------ | | [Add actor type and ID to logs ](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/838) | 1 | 18.11 | gitlab-shell | | [Pass key_id to /git_audit_event ](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/833) | 2 | 18.11 |gitlab-shell | | [Release v14.45.0 ](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/823) | 1 | 18.11 | gitlab-shell | | [Bump GitLab Shell to 14.45.0 ](https://gitlab.com/gitlab-org/gitlab/-/work_items/562581) | 1 | 18.11 |gitlab | #### Phase 4: Enable feature flag | Issue | Weight | Milestone |Repo |Done?| | ------ | ------ | -------| ------- |------ | | [Remove the human actor guard](https://gitlab.com/gitlab-org/gitlab/-/work_items/591573) |1|19.0|gitlab|| | [Enable log git streaming audit events FF](https://gitlab.com/gitlab-org/gitlab/-/work_items/415138) | 2 | 19.0 |gitlab | ### Testing Strategy After completing Part 1 (GitLab authentication fix): - Test HTTP-based Git operations with deploy tokens: - `git clone` over HTTPS with deploy token - `git pull` over HTTPS with deploy token - `git push` over HTTPS with deploy token - Test SSH-based Git operations with different authentication methods: - `git clone` over SSH (regular user, deploy token, SSH certificate, keyless SSH) - `git pull` over SSH (regular user, deploy token, SSH certificate, keyless SSH) - `git push` over SSH (regular user, deploy token, SSH certificate, keyless SSH) - Verify audit events are logged correctly for all actor types and protocols - Confirm no 404/401 errors in logs Once all tests pass: - Proceed with Part 2: Bump gitlab-shell, release, and enable feature flag - Monitor production during gradual rollout ## Ideal Outcome Git streaming audit events are safely rolled out to production, allowing GitLab to log all Git operations (clone, pull, push, fetch) with complete actor information, including regular users, deploy tokens, and SSH certificates. The feature flag is eventually removed, and the feature becomes a permanent part of GitLab's audit system. ## References * https://gitlab.com/gitlab-org/gitlab/-/issues/415138#note_2987600454 ## DRI (Best point of contact) * @emmaspark