Add Snowplow-compatible Duo events collector endpoint

What does this MR do and why?

Adds a Snowplow-compatible collector endpoint so a self-hosted AI gateway on an air-gapped instance has a local place to send Duo (DAP) billing events:

POST /api/v4/ai/events_collector/com.snowplowanalytics.snowplow/tp2

The endpoint is only reachable when the existing instance-level local_billing_persistence feature flag is enabled (the same flag that turns on local persistence in EE::Gitlab::BillingEvents::Client); otherwise it returns 404.

Authentication: the request must carry a personal access token that belongs to a service account and has the ai_features scope (api is also accepted, as on every endpoint). No token returns 401; a token with another scope, or one belonging to a human user, returns 403. The AI gateway sends the token through AIGW_BILLING_EVENT__API_KEY (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7288 (merged)). Granular (fine-grained) token authorization is deferred with route_setting :authorization, todo: until the dedicated fine-grained scope exists.

The endpoint accepts the Snowplow tracker protocol payload the AI gateway's BillingEventsClient emits. Ai::EventsCollector::IngestService decodes the billable_usage context and forwards it to Gitlab::BillingEvents::Client.track_billing_event, which already routes events on offline-licensed instances to Utilization::BillableUsage::RecordAggregateService.

Still to do in follow-up iterations (not in this MR):

  • Create the fine-grained global access scope for this endpoint and replace the todo: route setting with it.
  • Add API docs plus admin setup docs (service account and token creation).
  • Consider moving ingestion behind a Sidekiq job if batch sizes grow (RecordAggregateService notes DAP volume would need batching).

References

Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/631688

AI gateway counterpart: gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7288 (merged)

Screenshots or screen recordings

Not applicable, API-only change with no UI.

How to set up and validate locally

  1. Enable billing event tracking and local persistence:

    Feature.enable(:billing_event_tracking)
    Feature.enable(:local_billing_persistence)
  2. Create a service account and a token with the ai_features scope in the Rails console:

    sa = Users::ServiceAccounts::CreateService.new(User.admins.first, { organization_id: Organizations::Organization.first.id }).execute.payload[:user]
    token = PersonalAccessToken.create!(user: sa, name: 'aigw-collector', scopes: %w[ai_features], expires_at: 30.days.from_now, organization: sa.organization)
    puts token.token
  3. Send a Snowplow tracker protocol payload with the token. Replace NAMESPACE_ID with an existing group ID and TOKEN with the value printed above:

    CX=$(printf '{"schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1","data":[{"schema":"iglu:com.gitlab/billable_usage/jsonschema/1-0-3","data":{"event_id":"%s","event_type":"duo_agent_platform_workflow_completion","unit_of_measure":"tokens","quantity":5328,"timestamp":"2026-10-02T10:00:00","namespace_id":NAMESPACE_ID,"metadata":{"feature_qualified_name":"duo_agent_platform"}}}]}' "$(uuidgen)" | base64 | tr '+/' '-_' | tr -d '=\n')
    
    curl --request POST \
      --url "http://gdk.test:3000/api/v4/ai/events_collector/com.snowplowanalytics.snowplow/tp2" \
      --header "Authorization: Bearer TOKEN" \
      --header "Content-Type: application/json" \
      --data "{\"schema\":\"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4\",\"data\":[{\"e\":\"se\",\"se_ca\":\"DuoWorkflowService\",\"se_ac\":\"duo_agent_platform_workflow_completion\",\"aid\":\"gitlab_ai_gateway\",\"tv\":\"py-1.0.0\",\"p\":\"srv\",\"cx\":\"$CX\"}]}"
  4. Expect HTTP 200 with body {"status":"ok"} and a BillingEvents: entry in log/application_json.log. On an offline-licensed instance, Utilization::BillableUsage::DailyAggregate.last shows the aggregated row.

  5. Repeat without the Authorization header; expect HTTP 401. Repeat with a token of your own (human) user; expect HTTP 403 with Token must belong to a service account.

  6. Run Feature.disable(:local_billing_persistence) and repeat the request; expect HTTP 404.

Validate locally with the AIGW

Checkout to this MR follow the instruction on how to validate it locally and trigger a new DAP chat.

pipeline:skip-router-sync applied

The new were added routes in the cells https router repo gitlab-org/cells/http-router!1417 (merged) other routes are outputting the error.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Patrick Cyiza

Merge request reports

Loading
Loading