Skip to content

Added feature flag for disabling streaming for a group

Hitesh Raghuvanshirequested to merge
482997-disable-via-feature-flag into master

What does this MR do and why?

What?

We are introducing a feature flag disable_audit_event_streaming in this MR to disable audit event streaming to external destinations for a group.

Why?

In production incidents like gitlab-com/gl-infra/production#18489 (closed), sometimes the production load increases specially because of AuditEventStreamingWorker. Cause of these spikes is some malfunctioning external audit event destination which generally belongs to a single group. There are only 2 ways of handling such situations currently, one is by pausing the whole worker by which other groups also gets affected and other is by deleting the streaming destination record from rails console. Both these solutions are not optimal in terms of effect on wide no. of groups as well as deleting the records. In such instances the feature flag being added in this MR can help to disable the streaming of audit events for such group for sometime which can be later enabled.

We are also adding other measures to handle this situation in MRs !188491 (merged) and !186014 (closed) to reduce sidekiq retries.

This feature flag is short lived and will be removed once proper fixes are deployed on production.

References

How to set up and validate locally

  1. Create a group with Ultimate license.
  2. Add streaming audit event destinations to the group by following https://docs.gitlab.com/api/graphql/audit_event_streaming_groups/#add-a-new-streaming-destination.
  3. Now, make sure events of the group are being streamed to the streaming destination.
  4. Now disable the streaming of audit events for the group by running following command in rails console:
Feature.enable(:disable_audit_event_streaming, Group.find(<group_id>))
  1. Now try to make changes in settings of the group, so that audit events gets generated and ensure they are not being streamed to the external destination.
  2. Now again enable the streaming for the group and make sure audit events are being streamed.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #482997

Edited by Hitesh Raghuvanshi

Merge request reports