Improve experiment event validation process in staging/production

Created from:

Problem

Currently we are validating the tracking events during staging. We seem to not just be validating they are sent, but also trying to validate that they are structured in the expected way. We shouldn't need to validate the structure this late in the process and instead only need to validate the events are received in snowplow as expected.

Goals

  1. Improve validation of events for staging/production to be near realtime and accessible by all Growth.
  2. Move engineering ability to surface the raw events from development work before it gets to staging/production so that once we are in those later stages we are only verifying the end results in snowflake and not if we structured events properly, etc that should happen more to the left of the development cycle. This could overlap somewhat with #579137 and #579109

Possible solutions

realtime observance for staging/production - see ideas from @efergen in gitlab-org/growth/team-tasks#905 (comment 2854301979)

Edited by 🤖 GitLab Bot 🤖