GitLab Silent Mode for Backup / Disaster Recovery Testing

Introduction

In order to ensure that a backup has worked correctly, it's important to perform routine recovery testing. Likewise, when using GitLab Geo, for disaster recovery scenarios, a full recovery test helps ensure that procedures are working as expected.

However, when performing recovery testing on a GitLab backup, one of the issues faced is that the new "recovery test" environment may believe that it's production, and send out emails, webhooks, push mirrors, or communicate with connected Kubernetes servers (deprecated). This adds risk and complexity to recovery procedures. The usual approach to addressing this is to use firewall rules to block outbound traffic, but this has many problems:

1. It's easy to get firewall rules wrong, miss a required rule, leading to a "leak". This has lead to numerous incidents on GitLab.com, leading to customers receiving emails etc.
2. Outbound services may use Cloud Service APIs, which can be difficult to distinguish using firewall rules. Blocking the entire cloud service API may lead to unexpected results which may impact the recovery test.
3. Features such as SAML/LDAP/OAuth authentication may still require outbound connections and may require firewall changes, or be disabled, further complicating the testing
4. Dogfooding: all customers can benefit from the feature, including Standalone Omnibus and GET, GitLab Dedicated, and GitLab.com. Firewall rules are specific to a single instance.
5. Some features that perform outbound communications may fail and keep retrying, possibly interfering with the recovery test.

Proposal

1. Add a silent mode toggle (default off) in the GitLab application settings. This feature is enabled, no outbound communications will emanate from the GitLab instance, allowing recovery testing.
2. Any feature that performs outbound communications should check the application setting before performing the task, and silently and, when possible - successfully, return. This includes (but not limited to)
   1. Outbound Emails
   2. Outbound Webhooks
   3. Push Mirroring and pull mirroring
   4. Deprecated Kubernetes Connections (not needed for KAS)
   5. GitLab Alerting Notifications
   6. Slack webhooks, other services

Since these checks can be added at a fairly low level, it's likely that a small set of gateways within the codebase could provide total coverage.

Benefits

Having this toggle in place would allow for easier, less complex and more predictable automated recovery testing: something that is being discussed for GitLab Dedicated. It would also bring benefit for GitLab.com. Additionally, it would make recovery testing much easier for self-managed instances to perform backup and Geo DR recovery failover practice.

References

1. (Limited access) description of how GitLab Dedicated could use this feature to perform routine automated DR testing: https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/merge_requests/237#note_1058087852

cc @awthomas @joshlambert @glopezfernandez @marin @o-lluch @mkozono @juan-silva @jarv

added Category:Disaster Recovery gitlab.com production request + 1 deleted label

changed the description

changed the description

Outbound services may use Cloud Service APIs, which can be difficult to distinguish using firewall rules. Blocking the entire cloud service API may lead to unexpected results which may impact the recovery test. Features such as SAML/LDAP/OAuth authentication may still require outbound connections and may require firewall changes, or be disabled, further complicating the testing

Thanks for this proposal @andrewn . One common issue we have had limiting egress at the network layer is for outbound requests to object storage which may be worth calling out. I think it's possible to get IP ranges now for GCP https://www.gstatic.com/ipranges/cloud.json but I'm not sure if this includes storage buckets.

I know from my own experience trying to deploy GitLab in an isolated environment that the surface area is quite large so this will probably be challenging if we try to do it at the application.

One common issue we have had limiting egress at the network layer is for outbound requests to object storage which may be worth calling out.

Right. Handling this at the application layer might be a better option in this case too.

I know from my own experience trying to deploy GitLab in an isolated environment that the surface area is quite large so this will probably be challenging if we try to do it at the application.

This is to my point: either this is done in the application or every self-managed instance administrator, plus GitLab Dedicated, plus GitLab.com will need to handle this piecemeal. While it will take some effort to do it in the application, it'll be far greater effort to handle this in each different environment individually.

Handling this in the application pushes the problem left.

Thanks for the proposal, Andrew. This is a very interesting concept and I appreciate the benefits it would bring, especially around failover testing.

One question regarding:

it's likely that a small set of gateways within the codebase could provide total coverage

Are you saying that even though we would do it at the application level, we would do this in certain low-level components that handle outbound communication for a wide range of features in the system? How confident do you feel about this? Do you have some examples of what those gateways are and which teams own those areas of the codebase?

I think the point that John was maybe making is that there are so many areas that use outbound communication flows that it may not be a small set of gateways or the variety of use cases may make it challenging to handle the silent failure to communicate gracefully.

Nevertheless, a good approach to explore.

@juan-silva I haven't done an investigation into this, but if its the case that a particular type of outbound communication has many different potential flows, we should treat that as technical debt and use this opportunity to fix it.

What do I mean by this?

1. If the application has 6 different frameworks for sending an email, we might want to consolidate down to a handful as part of this exercise.
2. If we have more than a handful of Sidekiq jobs to perform a webhook push, perhaps we should consider consolidating these two.

Additionally: however much work it is for engineering to build these gates into the product, it's much, much more work for infrastructure to attempt to "catch" these outbound communications "at a network level" later on. With the "block the network" approach, every customer of GitLab that needs this, (including internal, GitLab.com and Dedicated) must work it out for themselves as nothing can be shared at the firewall level. By bringing this into a product, many GitLab users can benefit. This is where we see the benefits of dogfooding.

Great, thanks for the additional context Andrew. @sranasinghe FYI, this is the feature request issue I was talking about on our last check-in.

added automation:ml groupgeo + 1 deleted label

This issue was automatically tagged with the label groupgeo by TanukiStan, a machine learning classification model, with a probability of 1.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.

This message was generated automatically. You're welcome to improve it.

Add a silent mode toggle (default off) in the GitLab application settings. This feature is enabled, no outbound communications will emanate from the GitLab instance, allowing recovery testing.

Funnily enough, there was a long discussion in gitlab-foss#36073 (moved) and gitlab-foss#50020 (moved) which limited the scope of the initial proposal. I agree that having a way to silent the instance would be really useful.

marked this issue as related to #19052

marked this issue as related to gitlab-foss#50020 (moved)

marked this issue as related to #23585 (closed)

Setting label(s) devopssystems sectionenablement based on groupgeo.

added devopssystems sectioncore platform + 1 deleted label

👋 @andrewn, please add a type label and a subtype label to this issue to help with better milestone planning and issue discovery in issue reports. If this issue is not a feature, bug or maintenance then you may apply the type::ignore label to exclude it from type tracking metrics and future prompts.

---

(improve this comment?)

added auto updated + 1 deleted label

removed groupgeo + 1 deleted label

added groupgeo typefeature + 1 deleted label

added featureaddition + 1 deleted label

@sranasinghe For your consideration and as related to https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/merge_requests/237/

I am planning to do some investigation to get an idea of effort here, as long as https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/merge_requests/237/ is open or if it gets merged.

Thanks @mkozono that would be awesome.

I summarized current findings in #371816.

We've identified 7 features/items that should be blocked, at 1-2 weeks of effort each. And also 3 items of unknown effort at the moment, though I think we can count one or two of them as likely 1-2 weeks also.

Many of these items can be worked on in parallel, and perhaps other teams can contribute to some items within their domains, if we already have a silent_mode? method for them to call. Though it would probably fall on Geo to implement blocks in unowned code like Gitlab::HTTP or ActionMailer.

A caveat: In many cases, blocking outbound requests would e.g. result in 500s or Sidekiq failures, and it's not clear which of those might be a problem during a failover test, and how much additional work would be needed to handle those failures or adjust QA.

Oh, plus adding the application setting.

And Sampath had a great suggestion: Silent mode should be enabled during promotion because:

• The real primary should not be silenced.
• The promoted secondary should be silenced before it believes it is a primary, or else there is room for race conditions.

marked this issue as related to #371816

mentioned in issue #371816

Large Premium customer interested in this issue.

Use case 1: From test instances, no e-mail notifications should be sent for certain actions.

Use case 2: Not having all admins receive emails on production when e.g. the "user cap" has been exceeded which happens multiple times a day.

added customer + 1 deleted label

mentioned in issue #384170

marked this issue as related to #384170

This feature is a must!!!!

added workflowvalidation backlog + 2 deleted labels

@sranasinghe - GitLab Dedicated has a cross-dependency on this to deliver on our FY24 roadmap. I suggest you, @awthomas and I coordinate around the priority of this!

cc @marin @andrewn @o-lluch @juan-silva

mentioned in issue gitlab-org/geo-team/discussions#5056 (closed)

added theme:horse + 1 deleted label

Promoting this to an epic.

closed

promoted to epic &9826

Issue promoted to epic. Removing geoplanning label.

removed 1 deleted label