Draft: Define command palette AI answer lifecycle events and their join key

🤖 AI-authored change.

Declares six internal event definitions for the command palette AI answer lifecycle — YAML only, no emitters yet. Roughly 160 added lines across 6 new files under ee/config/events/. Every event carries search_request_id in the built-in property slot, so one answer's whole lifecycle reassembles with a single GROUP BY on se_pr and joins to the lexical search underneath it.

Action	What it records
request_palette_answer	Promote gesture requested an answer; opens the lifecycle
receive_palette_answer_first_token	First token painted (time to first token)
complete_palette_answer	Answer streamed to completion (terminal)
refuse_palette_answer	Answer refused rather than attempted (terminal)
degrade_palette_answer	Answer aborted/degraded on a latency or main-thread budget breach (terminal)
render_palette_answer_citation	One citation rendered; fires once per citation

What a reviewer should look at:

• The join key is minted client-side and depends on no feature flag. The only server-side minter, increment_search_counters, is reached from #show (app/controllers/search_controller.rb:211,273); the palette's rows come from #autocomplete (line 127), which never calls it, so no server-minted key exists on this surface with search_relevancy_join_key on or off. search_request_id is therefore minted in the client when the answer lifecycle opens, and an absent value means the client minted none rather than lost data. (Corrected at 352b97b6; earlier revisions of this description stated a dependency on that flag.)
• value is a shared warehouse column with a per-event clock origin. request_palette_answer measures from palette-open, the other four millisecond events from the promote gesture. degrade_palette_answer is milliseconds or a dimensionless ratio depending on label. Each description now states this; averaging se_va across the six is wrong.
• classification: duo is a data-collection choice, not a label. It makes these six eligible even when Snowplow is disabled and product usage data is off.
• Verification is validator-only. Schema validation plus a mutation table; no RSpec, no emitters, and the Snowplow Micro run predates the round-1 fixes so 2 of its 7 payloads no longer match their definitions.

🤖 Full mechanism, file:line citations and verification evidence (for agents / deep readers)

What is in the diff

Six new files, all new, no Ruby, no JavaScript, no specs:

• ee/config/events/request_palette_answer.yml (26 lines)
• ee/config/events/receive_palette_answer_first_token.yml (24 lines)
• ee/config/events/complete_palette_answer.yml (30 lines)
• ee/config/events/refuse_palette_answer.yml (24 lines)
• ee/config/events/degrade_palette_answer.yml (30 lines)
• ee/config/events/render_palette_answer_citation.yml (28 lines)

All six use classification: duo, product_group: global_search, product_categories: [global_search], milestone 19.4, tiers premium and ultimate, and identifiers user/namespace/project. Plus three later commits: two description-only wording changes and one that repointed introduced_by_url at this merge request.

These are declarations only. There are no emitters yet; the client-side code that fires these events is separate future work. An event definition with no producer is an established, tolerated state in this repo — 306 of the 1689 actions in this branch have no reference anywhere in the codebase (that count was taken in the branch, so it includes these six, which are unreferenced by construction).

The join key

Every one of the six carries search_request_id in the built-in property slot — ee/config/events/request_palette_answer.yml:19-20, and line 19 in each of the other five. The built-in slot means it travels as the flat structured-event field se_pr rather than inside gitlab_standard.extra.

The key is produced by merge request 253076, which introduces it in app/helpers/search_helper.rb behind the feature flag search_relevancy_join_key (gitlab_com_derisk, default off). It does not exist on master yet, so these definitions describe a key that arrives with that MR.

Why one event per citation rather than a list

The event pipeline whitelists only String, Integer and Float for custom properties — Gitlab::Tracking::EventValidator::CUSTOM_PROPERTIES_CLASSES at lib/gitlab/tracking/event_validator.rb:14, enforced at line 90. An array of citations therefore cannot be a custom property.

The cost is cardinality: N citation events per answer. The mitigation is citation_count on complete_palette_answer (ee/config/events/complete_palette_answer.yml:29-30) as the denominator, so a dropped citation event is detectable rather than invisible. That description also states the population trap: render_palette_answer_citation fires for citations painted on an answer that was later degraded (ee/config/events/render_palette_answer_citation.yml:2), so the numerator must be restricted by joining se_pr to a complete_palette_answer row or the rate mixes populations and can exceed 1.

Units and clock origins

value is a duration in milliseconds on five of the six events. On degrade_palette_answer the unit depends on the label (ee/config/events/degrade_palette_answer.yml:23-24): milliseconds for ttft, inter_token_stall and wall_clock_abort, and a dimensionless ratio for duty_cycle. inter_token_stall has no clock origin at all — it is a gap between two consecutive paints — and duty_cycle has no clock.

Every value description now states its unit, because all of them land in the same se_va warehouse column.

Units alone are not enough to make them comparable. request_palette_answer measures from the palette being opened (ee/config/events/request_palette_answer.yml:23-24); the other four millisecond events measure from the promote gesture, and its interval ends where theirs begin. The shared sentence therefore says the start point differs per event, and request_palette_answer states explicitly that it must not be averaged together with the others.

Why palette-open latency stays in value

The palette-open latency was left in value rather than moved to a named property. An earlier revision of this description justified that by saying a named property "would land in se_va either way". That was wrong, and this MR's own citation_ordinal fix depends on it being wrong: only label, property and value (BASE_ADDITIONAL_PROPERTIES, lib/gitlab/tracking/event_validator.rb:9-13) become se_la/se_pr/se_va (lib/gitlab/tracking/destinations/snowplow.rb:25-36); every other property is split off as extra (lib/gitlab/internal_events.rb:57-58,92-94) and lands in gitlab_standard.extra (lib/gitlab/tracking/standard_context.rb:24,54). A named palette_open_to_promote_ms would therefore leave se_va entirely.

The real reason it stays in value is that value is the only summable slot. The sole sum operator an event selection rule can declare is sum(value) (lib/gitlab/usage/event_selection_rule.rb:62), and update_sums returns early unless the :value key is present (lib/gitlab/internal_events.rb:103-111), so a duration held in a named property could never back a Redis-backed sum metric. Keeping it in value costs a shared column with a per-event start point, which the descriptions now state explicitly; moving it would cost the metric.

Round-1 review found the citation event was using value as an ordinal position, which would have mixed positions and durations in that one column. The ordinal moved to its own citation_ordinal Integer property (ee/config/events/render_palette_answer_citation.yml:25-26), and value on that event is now the milliseconds from the promote gesture to the citation being painted (lines 23-24).

Terminal-state exclusivity

Exactly one of complete_palette_answer, refuse_palette_answer and degrade_palette_answer fires per answer. Each of the three now states that contract in its own description — ee/config/events/complete_palette_answer.yml:2, ee/config/events/refuse_palette_answer.yml:2, ee/config/events/degrade_palette_answer.yml:2.

A degraded answer emits degrade_palette_answer and never complete_palette_answer, even when degrade_action is freeze_region or truncate_answer and a partial answer stays on screen. So token_count and citation_count are absent for degraded answers, and per-answer aggregates built on them cover completed answers only.

One overlap worth knowing before building rate metrics: a ttft budget breach produces both degrade_palette_answer with label ttft and receive_palette_answer_first_token with label over_budget for the same answer — but only when a first token was still painted (tokens_painted_before_degrade > 0, ee/config/events/degrade_palette_answer.yml:29-30). receive_palette_answer_first_token fires on paint, so a breach that aborts the stream before any token is painted emits no first-token event and there is nothing to de-duplicate. A breach-rate denominator counting both events double-counts the first case; de-duplicating unconditionally under-counts the second.

Runtime consequence of classification: duo

classification: duo (line 4 of each of the six files) makes duo_event? true (lib/gitlab/tracking/event_definition.rb:94-96), which places these six actions into EE::Gitlab::Tracking::EventEligibilityChecker.internal_duo_events (ee/lib/ee/gitlab/tracking/event_eligibility_checker.rb:33-37). eligible? therefore returns true for them even when Snowplow is disabled and product usage data is off, on any non-self-hosted instance.

That is the standard Duo posture for an AI surface, but it is a deliberate data-collection choice rather than a labelling one, so it is called out here.

Verification — what was actually run

• All six validate against the schema set used by the CI job internal-events-definitions-validation-ee (the standalone event-definitions-validator pinned at v1.2.1). Re-run after the wording edits: six green.
• A mutation table on request_palette_answer.yml confirms the validation harness can actually fail. 11 of 12 mutations are correctly rejected (the earlier "11 of 13" counted the unmodified baseline row as a mutation): bad product_group, missing milestone, malformed milestone, missing introduced_by_url, missing product_categories, an unknown top-level key, a bad tier, a capitalised action, an extra key inside an additional_property, a bad identifier, and a misspelt classification.
• One mutation passes and is expected to: a bogus product_categories value is accepted. This is not a limitation of the local harness — product_categories is not validated by CI at all. The schema annotates it with a custom format known_product_category, which appears exactly once in the whole event-definitions-validator project (lib/schemas/schema.json:75) with no registered handler, and json_schemer treats an unknown format annotation as a no-op. Confirmed by running the real Ruby validator 1.2.1: the mutation is green there too. global_search was checked by hand against config/feature_categories.yml:78.
• All six action names are globally unique across the 1683 pre-existing definitions (1689 in the branch, including these six), and each filename equals its action.
• Re-run at the current sha with the real Ruby validator (event-definitions-validator 1.2.1 under json_schemer 2.5.0, the version the CI component pins), not a Python approximation: six valid, exit 0. A 12-mutation regression table on top of it rejects 9 and accepts 3 — the product_categories value (unvalidated upstream, above) and the deletion of each of the two clock-origin / ttft clauses added in the latest commit. Those last two are the honest calibration: the wording fixes are prose, and nothing automated guards prose. Restore control green and byte-identical.
• Emission was exercised end to end through InternalEvents into a live Snowplow Micro collector at 608de433, before the round-1 fixes: seven events sent, seven arriving good, one distinct join key across them.

What I did not verify

• The Micro run does not cover the property set this MR now ships. render_palette_answer_citation's payload has changed since: value was the citation's ordinal position and is now a duration in milliseconds, and citation_ordinal is a new declared property. The run's own table shows those two rows emitting se_va=1 / se_va=2 — the ordinal, in the field now documented as milliseconds — and carrying no citation_ordinal at all. So 2 of the 7 emitted payloads no longer match their definitions, and citation_ordinal has never been emitted by anything. Re-running Micro is not a gate for a definitions-only MR with no emitters, but the coverage claim is scoped to the sha that earned it.
• The join key is verified in aggregate only. The Micro run's evidence table printed se_la, se_va and the extra payload per row, but did not print se_pr per row — the join key appears only as an aggregate "distinct join keys = 1" line. A run in which a single event dropped its property would have produced the same aggregate line. That per-row proof has not been re-run.
• No RSpec. These are YAML declarations; nothing in spec/ loads them outside a full Rails boot.
• Nothing about the client-side emitters, because they do not exist yet.
• product_categories: global_search is not machine-validated anywhere — only hand-checked against config/feature_categories.yml:78.

Review status

Went through the fleet's agent self-review loop before any human was asked: round 1 found one blocker and four improvements; round 2 found one blocker (a false verification claim), two improvements and a nitpick; round 3 found two blockers (an over-length commit subject that failed danger-review, and the false se_va justification corrected above) and a nitpick, all addressed. Round 4 converged. The MR is now out of Draft with @hsilva5 requested as the Analytics Instrumentation reviewer. Not to be merged by the fleet.

Changelog — review round 5 (fc2d2eb9a6e2)

Two GitLabDuo threads on the definitions, both about descriptions drifting from reality. Both fixed, both resolved.

• Baked budget numbers removed. receive_palette_answer_first_token's label no longer restates "600 ms at p95, derived from the measured Lane B p95 of 516.4 ms", and complete_palette_answer's max_inter_token_gap_ms no longer restates "the measured 400 ms inter-token stall budget". Both now describe the comparison qualitatively and say the value is deliberately not restated, because nothing ties this YAML to the constant the client-side guard will enforce. The live number still ships per row in degrade_palette_answer's budget_value.
• The flag-off join key is documented in the schema, not only here. All six property descriptions now state that search_request_id is absent when the search_relevancy_join_key feature flag is disabled, which is the default today, so a missing se_pr means the flag was off rather than a dropped property or lost data.

Re-verified at this sha with the real Ruby event_definition_validator 1.2.1: six files, "All files are valid", exit 0. A control run in the same session with milestone: '19.4.0' turns it red with a pattern error at /milestone.

Changelog — review round 6 (352b97b6, b39c0f26)

• The join key no longer depends on search_relevancy_join_key. increment_search_counters — the only server-side minter — is called from haml_search_results (app/controllers/search_controller.rb:211,273), which is on the #show path. The palette's rows come from #autocomplete (line 127), which never reaches it, so no server-minted key exists on this surface with the flag on or off. search_request_id is now minted client-side when the answer lifecycle opens, and all six property descriptions say so: an absent value means the client minted none, not lost data. This supersedes the flag-off wording added in round 5 and the dependency stated in the summary above.
• The four untested transformation blocks now have specs. rspec:undercoverage on pipeline 2863370450 flagged request_palette_answer, receive_palette_answer_first_token, refuse_palette_answer and render_palette_answer_citation at 50% with the search_request_id line at 0 hits. Each now has an emission context asserting the full extras hash. Label fixtures are taken from each definition's own documented vocabulary so the spec and the YAML cannot drift. Merged-results pipeline 2863763629 is green at b39c0f26.

Danger warnings not actionable here: three commits (352b97b6, 8f3543b2, c41f19b2) carry body lines over 72 characters. All three are already on the remote and under review, and shortening them needs a history rewrite and a force-push, which would discard the reviewer's diff anchors for a warning.

Parent epic: &23480