## Problem <!-- The problem being addressed by this issue --> In https://gitlab.com/groups/gitlab-org/-/epics/13223, the Analytics Instrumentation team made great strides by implementing a first iteration of standardized tracking in the AI Gateway. With Snowplow events now standardized and flowing into our data warehouse (specifically in the [wk_mart_behavior_structured_event_ai_gateway_flattened](https://dbt.gitlabdata.com/#!/model/model.gitlab_snowflake.wk_mart_behavior_structured_event_ai_gateway_flattened) table), there are a few follow-up actions needed to enhance the scalability, clarity, and overall success of this standard tracking system. Given that AI features are a top priority for the company, our tracking methodology should be aligned with the standards of other internal tracking systems. **Next Steps for Iteration:** **1. Add ‘feature’ value to event_label in the standard context** Context: The new AI Gateway tracking mechanism is set up to fire a 'request_unit_primitive' event which we then map to the 'features' (called 'services' in the [cloud_connector.yml](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml) file and renamed 'feature_name' in our [mapping datasource](https://dbt.gitlabdata.com/#!/model/model.gitlab_snowflake.cloud_connector_configuration)). So, a 'request_unit_primitive' event is triggered by a user at a fixed moment in time which we then display on AI Gateway Reporting allowing stakeholders to group by the unit primitive name (which is the event that fired) OR all of the features associated with that one unit primitive. This could be confusing because we don't know exactly which feature/service was used nor do we have a way of finding out - we just have the many to many relationship .yml file. Pictured below is an example of this structure. During a meeting on [Unit primitives structure/mapping for analytics](https://docs.google.com/document/d/19Vt0to6zdhFDpp8ucQ03XjUB1qVIppJOtjLV8MyxiAk/edit#heading=h.awoimyekn419) on 2024-10-09 @nmilojevic1 and @pjphillips discussed the idea of capturing a standard 'how/where' a unit primitive request was made which could be an interchangeable concept with 'feature'. **Next step:** Capture a feature/service value in the event_label.  **2. Introduce a ‘feature category’ value to event_property in the standard context** Context: The decomposition of unit primitives in AI Gateway (e.g., the chat primitive breaking down into ask_epic and ask_issue) increases complexity. To maintain consistent tracking across these decompositions, broad AI feature categories such as chat, code suggestions, security, or summarization should be defined by the Product Org. This will allow us to capture trends and behavior across broader AI categories, similar to how we group Service Ping events into Stage, Group, SMAU, and GMAU categories in .yml files. **Next step:** Collaborate with Product leadership to define and implement these categories. **3. Document the meaning of AI Gateway events for transparency** Context: Currently, there is no comprehensive documentation detailing the behaviors associated with AI Gateway events like `request_semantic_search_issue`. This limits understanding to PMs working directly with unit primitives, despite the fact that these events are intended to be used across all teams - Sales, Customer Success, Marketing, and Product. **Next step:** Create documentation explaining the meaning of these events for broader organizational use. **4. Document the technical details of AI Gateway event tracking in the handbook** Context: AI Gateway tracking is new to GitLab and partially automated. Engineers across different Stage::Groups working on AI features need a clear understanding of how this tracking system works, similar to our documentation for standard event tracking in Snowplow and Service Ping. **Next step:** Provide detailed technical documentation in the GitLab handbook. **5. Define a standard process for testing instrumentation and tracking of unit primitives** Context: AI Gateway’s tracking system differs from other standard tracking processes. PMs and Engineers must be able to verify that tracking behaves as expected when new unit primitives are added or when existing ones are decomposed. **Next step:** Document a process for testing instrumentation and event tracking. **6. Capture `delivery_type`, `is_internal` & `deployment_type` in the event payload instead of relying on mapping to Service Ping** Context: Not all SM installations send a service ping - approximately 25% of subscribed installations are opted out of reporting. Currently, `delivery_type`, `is_internal` & `deployment_type` are mapped on to installation product entities data only if a service ping is sent. [Link to reference to DBT SQL logic](https://dbt.gitlabdata.com/#!/model/model.gitlab_snowflake.wk_mart_behavior_structured_event_ai_gateway_flattened) This means that we can expect to see around 25% missing data for these dimensions. **Next step:** Capture `delivery_type`, `is_internal` & `deployment_type` in the event payload instead of relying on mapping to Service Ping ## Desired Outcome <!-- The desired state of the affected system after this issue has been completed --> Create an Epic with all next pieces of work represented as Issues and DRIs defined to work on next iteration ideally in Q4. ## Potential Solution <!-- An outline of potential solutions to get to the desired outcome. These solution(s) can still be adjusted throughout the implementation as long as the desired outcome is achieved. --> ### Documentation Update Needed <!-- Does this change require the documentation to be updated? check the appropriate option --> - [x] Yes - [ ] No ## How to verify <!-- How can we verify that the desired outcome has been achieved? The instructions from this section should be used to move the issue from ~“worfklow::verification”. --> ## Further actions needed <!-- Any further tasks that need to be completed after the main work of the issue is done, such as announcing the changes. --> <!-- This section is optional. --> <!-- Make sure to add one of the type labels (as per https://handbook.gitlab.com/handbook/engineering/metrics/#work-type-classification):--> <!-- /label gitlab~2278648 gitlab~10230929 ~"type::tooling" gitlab~15119514 -->