Gate AI audit event storage on the cascading setting

Andrew Jungrequested to merge
andrew-jung/604014-ai-audit-storage-model into master

What does this MR do and why?

Makes ai_audit_events_storage_enabled a working cascading setting (instance -> group -> project), has the audit pipeline honor it, and exposes it via GraphQL. Second and final backend MR for https://gitlab.com/gitlab-org/gitlab/-/work_items/603892, stacked on the columns MR.

Storage is disabled by default: customers must explicitly opt in before AI audit events (which can carry prompt content) are persisted. Streaming and logging are unaffected.

Model and services

  • Declares cascading_attr :ai_audit_events_storage_enabled on NamespaceSetting and ProjectSetting, with read delegates on Namespace and Project.
  • Write-time cascade per the current recommendation: the setting is registered in Ai::CascadeDuoSettingsService and in the group and application-settings update services, so a change enqueues the existing cascade workers and writes the value down the hierarchy.
  • Changing the setting requires the new update_storage_ai_audit_events permission, granted to Security Manager (Owner inherits it). This matches the rest of the AI-audit-events family (read_compliance_dashboard for the report, admin_external_audit_events for streaming destinations) and is stricter than the Maintainer-level admin_project the other project settings use - a maintainer should not be able to enable storing data they cannot view. Enforced in EE::NamespaceSettings::AssignAttributesService (group params), the projectSettingsUpdate mutation (hard deny), and an EE::Projects::UpdateService param strip mirroring the SAST carve-out.
  • New projects inherit the group value at creation (reset_ai_audit_events_storage_to_inherit_from_namespace, mirroring reset_duo_features_to_inherit_from_namespace), closing the gap between project creation and the next cascade run.
  • Changing the group value is audited (ai_audit_events_storage_enabled_updated).

Storage gate

Ai::DuoWorkflows::ProcessAuditEventsWorker skips persistence (Postgres and ClickHouse paths) for events whose resolved setting is off. The gate fails closed: if the event's project/namespace cannot be resolved, the event is not stored. Streaming and audit_json logging still happen either way.

GraphQL

  • Group.aiAuditEventsStorageEnabled + Group.lockAiAuditEventsStorageEnabled (read), settable via groupUpdate.
  • ProjectSetting.aiAuditEventsStorageEnabled (read), settable via projectSettingsUpdate.
  • All fields/args are experiment: { milestone: '19.2' }. The project value follows the ProjectSetting-nested pattern (toolApprovalForSessionEnabled precedent), not the older flat-field-on-Project duplication.

Setting semantics (value + lock)

The value is the default for descendants; the lock is the mandate. Same model as duo_features_enabled:

Group value Group lock Result for projects beneath
Disabled Unlocked Projects default to off, may opt in
Disabled Locked Off everywhere, projects cannot enable
Enabled Unlocked Projects default to on, may opt out
Enabled Locked On everywhere, projects cannot disable

The same relationship applies instance -> group. Group value writes cascade to all descendants (overwriting prior project choices, like the other Duo settings); lock enforcement is read-time and immediate; new projects copy the group value at creation. A frontend can present fewer of these states than the API supports.

Feature flag

Everything is behind enforce_ai_audit_events_storage_setting (default off): with the flag off, the worker stores as before and the group/project write paths strip the params, so production behavior does not change until rollout.

Database queries

New SQL introduced by this MR:

  1. Project creation sync (reset_ai_audit_events_storage_to_inherit_from_namespace) - single-row update by unique-indexed FK, only when the group value is true:

    UPDATE "project_settings" SET "ai_audit_events_storage_enabled" = TRUE WHERE "project_settings"."project_id" = 123

  2. Worker gate lookups - primary-key Project.find_by_id / Namespace.find_by_id, memoized once per [project_id, namespace_id] scope per batch.

The cascade introduces no new query shapes: it adds one column to the SET clause of the existing Ai::CascadeDuoSettingsService batched updates (NamespaceSetting.for_namespaces(ids).update_all / ProjectSetting.for_projects(ids).update_all, batches of 100 via NamespaceEachBatch), which were introduced and database-reviewed with the Duo settings cascade. The group and application-settings update services only strip params when the feature flag is off (no SQL).

References

How to set up and validate locally

1. Enable the feature flag

# rails console
Feature.enable(:enforce_ai_audit_events_storage_setting)

# or scope it to a single actor:
Feature.enable(:enforce_ai_audit_events_storage_setting, Project.find_by_full_path('<group>/<project>'))

2. Set the setting through the API

With a PAT (api scope), as a group/project owner or security manager (update_storage_ai_audit_events). Group value + lock:

curl -s "http://gdk.test:3000/api/graphql" \
  -H "PRIVATE-TOKEN: $TOKEN" -H "Content-Type: application/json" \
  -d '{"query":"mutation { groupUpdate(input: { fullPath: \"<group-path>\", aiAuditEventsStorageEnabled: true, lockAiAuditEventsStorageEnabled: true }) { group { aiAuditEventsStorageEnabled lockAiAuditEventsStorageEnabled } errors } }"}'

Project:

curl -s "http://gdk.test:3000/api/graphql" \
  -H "PRIVATE-TOKEN: $TOKEN" -H "Content-Type: application/json" \
  -d '{"query":"mutation { projectSettingsUpdate(input: { fullPath: \"<group-path>/<project-path>\", aiAuditEventsStorageEnabled: true }) { projectSettings { aiAuditEventsStorageEnabled } errors } }"}'

Read the group value back with { group(fullPath: "<group-path>") { aiAuditEventsStorageEnabled lockAiAuditEventsStorageEnabled } }. The project value is readable via console (project.ai_audit_events_storage_enabled) — there is no query field on Project by design, matching the toolApprovalForSessionEnabled precedent.

With the flag disabled, both mutations strip the argument: the group mutation leaves the setting unchanged, and the project mutation returns Must provide at least one argument when it was the only argument.

3. Verify events are not stored (PostgreSQL and ClickHouse)

Simulate what the ingest pipeline enqueues, with the setting at its default (off):

# rails console
user     = User.find_by(username: 'root')
project  = Project.find_by_full_path('<group>/<project>')
workflow = Ai::DuoWorkflows::Workflow.create!(user: user, project: project)

event = {
  'cloud_event_id' => SecureRandom.uuid,
  'event_name'     => 'ai_llm_input_sent',
  'created_at'     => Time.current.iso8601,
  'author_id'      => user.id,
  'project_id'     => project.id,
  'namespace_id'   => nil,
  'ip_address'     => nil,
  'workflow_id'    => workflow.id,
  'details'        => { 'model' => 'claude' }
}

project.ai_audit_events_storage_enabled # => false (disabled by default)
Ai::DuoWorkflows::ProcessAuditEventsWorker.new.perform([event])

# PostgreSQL - no row:
AuditEvents::AiAuditEvent.exists?(cloud_event_id: event['cloud_event_id']) # => false

# ClickHouse (when use_clickhouse_for_analytics is enabled) - no row:
ClickHouse::DumpWriteBufferWorker.new.perform('ai_audit_events')
ClickHouse::Client.select(
  "SELECT count() AS c FROM ai_audit_events WHERE id = '#{event['cloud_event_id']}'", :main
) # => [{"c"=>0}]

The worker writes to ClickHouse when Gitlab::ClickHouse.globally_enabled_for_analytics? is true, otherwise to PostgreSQL — check whichever applies to your GDK.

Then opt in (mutation from step 2, or project.project_setting.update!(ai_audit_events_storage_enabled: true)), rerun the worker with a fresh cloud_event_id, and the same checks return the row (exists? => true on the PostgreSQL path, [{"c"=>1}] on the ClickHouse path). Opting back out gates new events again. Streaming and log/audit_json.log receive every event in all cases — only storage is gated.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Andrew Jung

Merge request reports