Skip to content

2024-04-04: Snowplow Low good events record volume

Customer Impact

None at the moment.

Detection

Snowplow low event count alerts were firing immediately

Current Status

Snowplow is collecting a normal level of events, but when enriching them, most appear to be marked as bad events.

Root Cause

A new field was added to the snowplow standard context, but the rails app wasn’t updated to reference the new schema version instead only the field was added in gitlab-org/gitlab!148463 (merged).

Affected events: All events sent by Gitlab.com directly (Frontend and Backend) since those include the standard context Unaffected events: Events sent by Editor Extensions, AI gateway or customers.gitlab.com, since those normally do not include the standard context.

Timeline (UTC)

  • 2024-04-04 20:41: gitlab-org/gitlab!148463 (merged) is deployed to production
  • 2024-04-04 20:43: First alert fires in #g_monitor_analytics_instrumentation
  • 2024-04-04 20:47: This incident is created
  • 2024-04-04 21:49: Fix MR gitlab-org/gitlab!148728 (merged) is requested to be merged, which adjusts the instance to use the right standard context version
  • 2024-04-04 23:49: Fix MR gitlab-org/gitlab!148728 (merged) is completely reviewed and set to be merged.
  • 2024-04-05 - 2024-04-08: Unfortunately fix MR is not deployed to production due to family and friends day.
  • 2024-04-08 06:59: Temporary fix https://gitlab.com/gitlab-org/iglu/-/merge_requests/114 is added to schema repository to add the missing field to previous version of schema in order to resolve the problem immediately.

📚 References and helpful links

Recent Events (available internally only):

Deployment Guidance

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.

Edited by Sebastian Rehm