[FF] snowplow_emitter_http_timeout -- Add HTTP timeout to Snowplow emitter

Summary

This issue is to roll out the feature on production, that is currently behind the snowplow_emitter_http_timeout feature flag.

Owners

  • Most appropriate Slack channel to reach out to: #g_analytics_analytics_instrumentation
  • Best individual to reach out to: @michold

Expectations

What are we expecting to happen?

When enabled, the Snowplow emitter will use a custom SnowplowTimeoutEmitter (subclass of AsyncEmitter) that sets open_timeout and read_timeout to 30 seconds on HTTP POST requests. This is to debug staging issues where event emission metrics don't match.

What can go wrong and how would we detect it?

Snowplow events may fail to send if the timeout is too aggressive. Monitor the gitlab_snowplow_failed_events_total and gitlab_snowplow_successful_events_total counters on https://dashboards.gitlab.net.

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

  • Verify the MR with the feature flag is merged to master and has been deployed to non-production environments with /chatops run auto_deploy status <merge-commit-of-your-feature>
  • Enable the feature globally on non-production environments with /chatops run feature set snowplow_emitter_http_timeout true --dev --pre --staging --staging-ref
  • Verify that the feature works as expected.
  • If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking deployments.

Before production rollout

  • Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production.

Global rollout on production

For visibility, all /chatops commands that target production must be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel.

Release the feature

  • Create a merge request to remove the snowplow_emitter_http_timeout feature flag.
  • Close this rollout issue.

Rollback Steps

  • This feature can be disabled on production by running the following Chatops command:
/chatops run feature set snowplow_emitter_http_timeout false
  • Disable the feature flag on non-production environments:
/chatops run feature set snowplow_emitter_http_timeout false --dev --pre --staging --staging-ref
  • Delete feature flag from all environments:
/chatops run feature delete snowplow_emitter_http_timeout --dev --pre --staging --staging-ref --production