### Problems The use of feature flags in Gitaly is currently restricted due to technical limitations: * We cannot use feature flags in background jobs because we always extract feature flag states from the RPC context. * We cannot use scoped feature flags. Those two problems came from the same root cause that feature flags are now propagated to Gitaly indirectly via workhorse and gitlab-shell, and directly via gRPC metadata. In either ways, Gitaly depends on user-initiated actions. When GitLab receives git-related requests, all flows go through GitLab rails and the feature flags are being evaluated there. Gitaly is incapable of querying the feature flags itself. The actors passed in pre-evaluation are also inconsistent between different locations. The detailed analysis can be found [here](https://gitlab.com/gitlab-org/gitaly/-/issues/4459#note_1134587493). ### Proposal In the current flow, Gitlab Rails is responsible for evaluating all feature flags, as it holds both flag definitions and current actor gate data. It makes a perfect sense to keep Rails as the centralized feature flag service. In a further future, if we are allowed to make a more aggressive move, we can replace the role of GitLab rails by another service. ```mermaid graph TD GL[Gitlab Rails] GS[Gitlab Shell] W[Workhorse] subgraph Gitaly Process G[Gitaly gRPC Server] BG[Background Jobs] FFA[FF Evaluator] FFR[FF Refresher] end BG -- "FeatureA.IsEnabled(ctx, repo)" --> FFA FFR -- "fetch feature gates via HTTP" --> GL FFR -- "refresh data" --> FFA GL -- return features via internal APIs --> W GL -- return features via internal APIs --> GS GS -- "RPC(gitaly-feature-a: true)" --> G W -- "RPC(gitaly-feature-a: true)" --> G GL -- "RPC(gitaly-feature-a: true)" --> G GL -- "collect actors" --> GL ``` #### Introduce Repository actor This is a prerequisite of the whole epic. From Gitaly's standpoint, current [supported actors](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors) are irrelevant. The main resource under management is Git Repository. It does not keep any reference or lookup hint to trace back a User/Group from a repository. In fact, Gitaly is designed with the ability to run independent of Gitlab in mind. As a result, we should introduce a new type of actor, dedicated to Gitaly. Obvious, that would be Repository. Repository's relative paths can be used as the unique identity for rolling out. In https://gitlab.com/groups/gitlab-org/-/epics/8903+, we look forward to coming up with a unique generated ID for each repository. When it's ready, we can switch to use that ID instead. From GitLab Rails side, a feature flag can be turned on for a repository like the following: ```ruby project = Project.find_by_full_path("gitlab-org/gitlab") Feature.enable(:gitaly_mep_mep, project.repository) # Or if the target is Project wiki Feature.enable(:gitaly_mep_mep, project.wiki.repository) # This repository actor also works with other types of repositorys Feature.enable(:gitaly_mep_mep, PersonalSnippet.first.repository) ``` The data stored in the feature gate table in Rails looks something like the following:  From Gitaly side, the feature flag evaluation can be fetched from gRPC metadata context or directly in background jobs: ```go if featureflag.GoFindLicense.IsEnabled(ctx, nil) { // Use pre-evaluated flags if available } if featureFlag.GoFindLicense.IsEnabled(ctx, repo) { // Invoke feature flag evaluation } ``` #### Propagate feature flag actors consistently in user-initiated actions As mentioned in the above section, feature flags actors are collected differently across evaluation locations. Before hitting Gitaly via gRPC, all flows must hit Gitlab rails for authentication and authorization. The feature flag overhead is minimal as the evaluation is conducted during the way. If we change this flow, there would be a chance that a random RPC results in multiple HTTP requests, regardless of how careful we cache. Therefore, we don't want to change the overall architecture because that way is riskier and may yield unexpected negative performance impact. Instead, we should focus on fixing the consistencies to ensure the actors are collected fully and accurately in all foreseen cases. This MR demonstrates this goal better: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101218+ #### Implement feature flag HTTP-based feature flag evaluator in Gitaly for Gitaly background jobs Sub-epic: https://gitlab.com/groups/gitlab-org/-/epics/9123+ Outside user-initiated RPC contexts, the only way for Gitaly to check a feature flag gate is to contact GitLab Rails web servers. Obviously, Gitaly needs to trigger HTTP requests to Rails web servers. However, it should not query the open status of a flag for an actor individually. Doing that results in a massive amount of HTTP requests, scaling linearly with the number of flags we used in Gitaly. Instead, we could implement client-side evaluation inside Gitaly. Flipper Ruby GDK supports this strategy officially: [Flipper HTTP](https://www.flippercloud.io/docs/adapters/http) and [Flipper API](https://www.flippercloud.io/docs/api). From [an analysis](https://gitlab.com/gitlab-org/gitaly/-/issues/4459#note_1134587493), Flipper - underlying FF engine - manages the status of a FF by using feature gates. The format of feature dates for a particular flag is straight-forward, especially for a flag already turned on globally. As soon as Gitaly has access to those feature gates, it can evaluate the status of a flag itself, following a fairly simple set of logic. The feature gates are also highly cache-able. Typically, the gates stay static for a significant amount of time, even for rolling out by a percentage of actors. Only human intervention can make the gates change. Even so, we generally accept a flag change takes time to fully propagate to all Gitaly nodes. That also means Gitaly nodes can accept the evaluation result from stale feature gates in case FF server is uncontactable. Gitaly also doesn't need to support feature gate modification. We (GitLab.com) advocates controlling the flags via chatops (via Slack). In some less common cases, we use Rails console. When deliver releases to customers, the flag's default values are controlled by YAML files. Customers can enable/disable the flags using consoles or via Admin APIs. Therefore, no need to modify the gates in Gitaly. Read-only evaluator is good enough for this current design. Although this design targets Gitaly background jobs, it can be expanded to gRPC servers and even other Gitlab components. Applying it everywhere in Gitaly sounds like a risky move. I would like to make a baby step first. Background jobs are not as time-sensitive as serving RPC requests. This epic is also a preparation to rollout https://gitlab.com/groups/gitlab-org/-/epics/8175+ in a safe fashion. ##### Design From the above points, I think we can design a fairly efficient client-side evaluator for Gitaly: * The evaluator fetches the definition and feature gates from Rails. It uses those data for evaluation. It does not trigger a flag to evaluate flags individually. The data fetched from Rails looked something like following: ``` { "gitaly_mep_mep": [ { "key": "actor", "value": "Repository:@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b"}, { "key": "actor", "value": "Repository:@snippet/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767"} ], "gitaly_feature_a": [ { "key": "percentage_of_actors", "value": "25"} ], "gitaly_feature_b": [ { "key": "boolean", "value": "true"} ] } ``` * The evaluator is non-blocking, as we would rather not add overhead to each flag invocation. * Code paths before/after a feature flag is enabled are required to be compatible. It means after turning the flag on, the new code path should not modify the data, causing it's impossible to run when turning the flag back. * Before initialization, the evaluator initializes flag data based on flag definition ([example](https://gitlab.com/gitlab-org/gitaly/-/blob/53ed8f3aec468482c63b0e9f341bfd35955b2bbd/internal/metadata/featureflag/ff_go_find_license.go#L8-8)). Each definition has a default value used as the last fallback method. * Evaluator requires ctx as the first argument. No need to re-evaluate the flag if it was evaluated somewhere else (user-initiated RPC requests). * All goroutines share the same in-process cache. It's likely we can use a singleton evaluator. Cache expired time is non-determined, maybe 15 mins. The evaluator should return the result immediately, based on available data. * The evaluator spawns a long-lived refresh goroutine. It actively fetches all feature gates in batch from FF server half-way prior to the expiration time. We can also add randomized delay to prevent all Gitaly nodes performs the same query at the same time. The HTTP requests should apply [ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag), of course the FF server must support it beforehand. ##### Failure modes * If the FF server is never uncontactable, FF evaluator uses the pre-defined default value in flag definition. * If the FF server is uncontactable occasionally, the refresh goroutine should retry continuously with exponential back off. During so, FF invocation still works with stale data. As explained above, I think it's perfectly fine to use stale data while waiting for a fresh data. After cache data are expired and the FF server is not back on, FF evaluator fallbacks to default value. ##### Performance impact * Following this design, the evaluator should not add noticeable overheads to Gitaly code paths, if goroutines are not blocked reading cache data. * For self-managed instances, there is likely only one Gitaly node. That node occasionally trigger HTTP request to internal FF server. This won't affect the activities of Rails web server. * For GitLab.com, we have more than 100 Gitaly nodes. If all nodes start at the same time, there should be a wave of FF requests to Rails web server. That's unlikely the case. Even so, the number of API pods are double the amount of Gitaly nodes. Adding sequential well-cached requests with Etag per some minutes per node won't affect the API fleet as well. ### Goals - [x] %"15.6" Fully support scoped feature flags for user-initiated actions with 3 types of actors: User, Repository, and Group - [x] %"15.6" Allow to enable feature flags via chat ops - [x] %"15.6" Scoped by individual actor, or by a percentage of actor - [x] %"15.7" Support scoped feature flags for Gitaly's independent jobs via 1 type of actor: Repository ### Side notes * There are two "Feature Flag" at GitLab. One is the feature flag used for internal Gitlab development - powered by [flipper](https://github.com/jnunemaker/flipper) ([doc](https://docs.gitlab.com/ee/development/feature_flags/index.html)). The other one is Feature Flag product we offer to customers - powered by [unleashed](https://www.getunleash.io/) ([doc](https://docs.gitlab.com/ee/operations/feature_flags.html)). All the Feature Flag terms mentioned in this epic infer the prior one, powered by Flipper. There are ideas about dogfooding the FF product at GitLab. However, I don't think we should do it at this stage because we'll end up with two different FF systems, co-exist in the same codebase. If we switch everything to our product (okay, technically, Unleashed is not owned by us), we'll need to rework all the corresponding tool-chains. The internals of two systems are also different. That said, when we tried to make a more aggressive move in the future, we can switch the underlying engine without changing the overall architecture. * There are also ideas (https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/issues/12+ for example) about implementing feature flags in Labkit so that all components can be beneficial from it. That's a good idea. However, I'm not sure which components can re-use such logic. Codes in labkit also requires a certain level of abstraction. Working in it also adds up significant overhead. The current design already involves multiple components. So, I would rather implement in the components directly. When the flow becomes mature, we can extract them to labkit. * In a further future, after the FF evaluator is stable, we can force Gitaly gRPC server to use it. Pre-evaluation inside Rails and context propagation can be dropped. <details><summary>A further future</summary> ```mermaid graph TD GL[Gitlab Rails] GS[Gitlab Shell] W[Workhorse] subgraph Gitaly Process G[Gitaly gRPC Server] BG[Background Jobs] FFA[FF Evaluator] FFR[FF Refresher] end BG -- "FeatureA.IsEnabled(ctx, repo)" --> FFA FFR -- "fetch feature gates via HTTP" --> GL FFR -- "refresh data" --> FFA G -- "FeatureA.IsEnabled(ctx, repo)" --> FFA GS -- "RPC" --> G W -- "RPC" --> G GL -- "RPC" --> G ``` </details> ### References and related works * https://gitlab.com/gitlab-org/gitlab/-/issues/217490+ * https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/issues/12+ * `@jcaigitlab` had a POC about calling feature flag in Praefect: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4584 * https://gitlab.com/groups/gitlab-org/-/epics/5325+ * https://gitlab.com/gitlab-org/gitlab/-/merge_requests/100458+ * Incidents that can be prevented due if we can roll out gradually by actor: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7864+ * https://gitlab.com/gitlab-org/gitaly/-/issues/3414+ ### Status 2020-11-10 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101218+ MR was merged. That MR laid the foundation to propagate the actors to Gitaly. Due to its complexity, I took a defensive step to roll it out: * Apply the propagation to Commit service only. We'll need to apply it to other Gitaly services in Gitaly. This work is tracked in https://gitlab.com/gitlab-org/gitaly/-/issues/4613+. * Hide the propagation behind a feature flag: https://gitlab.com/gitlab-org/gitlab/-/issues/381516+. Thereafter, the actors will be ready to use. I'll also need to update the documentation regarding this change.