Add feature flags to SnowPlow events
What
From request events, add the following instrumentation data to legacy.snowplow_unnested_events_30:
- json.feature_flag_states
- json.db_*
- json.graphql.*
- json.mem_*
- json.redis_*
- json.controller
- json.duration_s
- json.exception.class
- json.gitally_calls
- json.gitally_duration_s
Why
20% of gitlab.com events are logging the feature flags seen in that request (about 160k events/hr https://log.gprd.gitlab.net/goto/5fb6bfb0-e807-11ec-8656-f5f2137823ba -- internal link). This information can be very valuable, but it is currently only accessible through kibana, limiting its usability. By adding this data and the metrics collected on the events to snowplow, we can create dashboards and enable full observability over code changes. Some questions that this will enable us to answer:
- How many calls to the database this change adds?
- How much time does this feature impact on the 99% request time?
- How many users are seeing this feature flag?
- How much additional memory is this feature requesting?
Beyond question, these can also become alerts, or improve root cause analysis: "We are experience an unexpected hit on performance, which feature additions could be the cause of it?"