[Feature flag] Enable quarantined RPCs

What

Enable the :gitaly_quarantine feature flag, which enables the use of quarantined RPCs in the Operations service. This should speed up access checks given that Rails will switch over to using the quarantine directory.

Owners

• Team: Gitaly
• Most appropriate slack channel to reach out to: #g_create_gitaly
• Best individual to reach out to: pks-t

Expectations

What release does this feature occur in first?

%14.2

What are we expecting to happen?

by using the quarantine directory, all new objects get created in a separate object directory. This staging environment is then passed on to Rails' access checks, which know to do optimized RPC calls which use the quarantine directory in order to enumerate new Git objects. This is a lot more efficient compared to the unquarantined RPCs. Furthermore, we will now discard any new objects in case they aren't accepted by the access checks.

What might happen if this goes wrong?

Access checks start to misbehave. Objects don't end up in the main repo.

What can we monitor to detect problems with this?

• Feature status for the following RPCs should be monitored:
  • UserUpdateBranch
  • UserCreateBranch
  • UserCherryPick
  • UserCommitFiles
  • UserMergeToRef
  • UserMergeBranch
  • UserRevert
  • UserUpdateSubmodule

Roll Out Steps

• Read the documentation of feature flags
• Add featureflagstaging to this issue (howto)
• Is the required code deployed? (howto)
• Do we need to create a change management issue? (howto)
• Enable on staging (howto)
• Test on staging (howto)
• Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Announce on this issue an estimated time this will be enabled on GitLab.com
• Add featureflagproduction to this issue
• Enable on GitLab.com by running chatops command in #production (howto)
• Cross post chatops slack command to #support_gitlab-com and in your team channel
• Verify the feature flag is being used by checking Prometheus metric gitaly_feature_flag_checks_total
• Announce on the issue that the flag has been enabled
• Did you set the feature to both 100% and true (howto)
• Submit a MR to have the feature OnByDefault: true and add changelog entry (howto)
• Have that MR merged
• Possibly wait for at least one deployment cycle (howto)
• Submit an MR to remove the pre-feature code from the codebase and add changelog entry (howto)
• Have that MR merged
• Remove the feature flag via chatops (howto)
• Close this issue