## Summary Since the `code_completion_v4_endpoint` feature flag rollout, code suggestion usage events (`code_suggestion_shown_in_ide` / `accepted` / `rejected`) no longer include `language`, `model_name`, or `model_engine`. On GitLab.com, all code suggestion telemetry since 2026-07-07 (100% rollout) is affected. Most visible symptom: the **GitLab Duo Code Suggestions acceptance by language** chart on the Duo and SDLC trends dashboard is empty, because every event is bucketed under an empty-string language. ## Root cause 1. gitlab-org/gitlab!239271 routes code completions from the AI Gateway's legacy `/v2/code/completions` endpoint to `/v4/code/suggestions`, behind the `code_completion_v4_endpoint` flag (`gitlab_com_derisk`, milestone 19.1). 2. **Response shape difference**: v2 returns model metadata at the top level (`{"model": {"engine", "name", "lang"}, "choices": [...]}`), while v4 nests it under `metadata.model` (see the `CompletionResponse(choices: ..., metadata: ResponseMetadataBase(model: ModelMetadata(...)))` construction in `ai_gateway/code_suggestions/handler.py` in the ai-assist repo). 3. The monolith proxies the AI Gateway response to clients **verbatim** via Workhorse `send_url` (`ee/lib/api/code_suggestions.rb`), so the v4 shape reaches the IDE clients unchanged. 4. The language server reads the top-level field: `model: suggestionsResponse?.model` in `src/common/suggestion/suggestion_service.ts`, and the instance tracker sends `language: model?.lang`, `model_name: model?.name`, `model_engine: model?.engine` (`src/common/tracking/code_suggestions/code_suggestions_instance_tracker.ts`). With the v4 shape, `model` is `undefined`, so these properties are silently omitted from the `additional_properties` posted to `/api/v4/usage_data/track_event`. 5. `Gitlab::Tracking::AiTracking` therefore stores event extras without `language` → ClickHouse `ai_usage_events.extras` lacks the key → `language = ''` in `ai_code_suggestions` and `code_suggestion_events_daily` → the dashboard chart drops the empty-string bucket and renders its empty state. ## Evidence (gitlab-org/gitlab project, production data via the `duoCodeSuggestions` GraphQL aggregation) * 2026-05-01 → 2026-06-01: languages populated normally (ruby 7,931 shown / js 1,580 / go / python / ts) * Week of 2026-06-15 → 06-22: transition — 1,892 shown events with `language: ""` vs ~33 with a real language * 2026-06-22 onward: 100% of events have `language: ""` * `ide_name` remains fully populated in the same window (VS Code, Neovim, RubyMine, Web IDE), confirming the telemetry pipeline itself works and only the model-derived fields are lost ## Timeline correlation (feature-flag-log) | Date | Action | |---|---| | 2026-06-15 | flag set to `true` scoped to `feature_group` on gprd — matches exactly when gitlab-org/gitlab project data lost language | | 2026-06-25 | 5% of actors on gprd | | 2026-07-06 | 25% of actors on gprd | | 2026-07-07 | `true` (100%) on gprd | ## Impact * `language`, `model_name`, and `model_engine` are missing from all code suggestion events captured while the flag is enabled; the gap **cannot be backfilled** because the values were never recorded. * The acceptance-by-language chart on Duo and SDLC trends dashboards is empty for affected windows, and per-language filtering of AI usage analytics degrades equally (instance events path). * The language server's Snowplow tracker falls back to file-extension-based language detection, so the Snowplow stream retains `language` but still loses `model_name` / `model_engine`. * Note: the by-language chart was separately migrated to the aggregation engine (merged 2026-06-25). That migration is **not** the cause — the underlying events lack language regardless of the read path; the timing is coincidental. ## Proposed fix A server-side backward-compatibility fix restores telemetry for all existing clients immediately: * **Option A**: AI Gateway `/v4/code/suggestions` also returns a top-level `model` object matching the v2 contract (compat shim), or * **Option B**: the monolith re-shapes the v4 response back to the v2 contract before proxying to clients. A language-server-side fix (read `metadata.model ?? model`) is also worth doing long-term, but it only takes effect as IDE extensions/language server versions roll out to users, which would leave an ongoing data gap for older clients. Reverting/disabling `code_completion_v4_endpoint` would also stop the data loss while a fix ships. ## Links * Feature issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/598413 * Rollout work item: https://gitlab.com/gitlab-org/gitlab/-/work_items/602067 * Introducing MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239271 * Flag log: https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/59949, https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/60439, https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/60825, https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/60903