Fully support feature flag actors in Gitaly (!101218) · Merge requests · GitLab.org / GitLab

Quang-Minh Nguyen requested to merge qmnguyen0711/4459-allow-actor-limited-feature-flags into master Oct 17, 2022

What does this MR do and why?

Feature flags in Gitaly was in a weird state. GitLab Rails pre-evaluates the flags, then passes to Gitaly via gRPC metadata or indirectly through internal APIs and workhorse. When evaluating gRPC metadata, feature flags are not evaluated with any actors. In other words, all feature flags are global in the context of RPCs. In contrast, feature flags in other locations support actors partially—only Project actor is allowed. This inconsistency creates plenty of troubles while delivering Gitaly changes.

This commit enhances the feature flag situation by:

Introduce Repository actor, dedicated for Gitaly. This actor uses a repository relative path as its unique identity.

project = Project.find_by_full_path("gitlab-org/gitlab")
Feature.enable(:gitaly_mep_mep, project.repository)

Implement a collector in Gitaly client to collect possible actors for pre-evaluation
Apply actor collector to Gitaly CommitService as a POC for this approach

What happens after this MR is merged? Properly nothing changes. At the time of writing, all feature flags used in Gitaly were either not enabled, or rolled out globally. In both cases, this change does not affect them. We'll observe the performance impact for a reasonable amount of time before moving forward and apply the new mechanism to other RPCs.

List of supported actor types

Actor type	Scope
User	Users who are performing the current action. The user is inferred from authentication info. Some examples: - Signed-in users if the action is performed on UI - Owners of SSH public keys used when cloning via SSH - Owners of Personal Tokens used when querying an API<br>- etc.
Repository	Particular repositories managed by Gitaly
Group	All project repositories under the scopes of a group

When a flag is enabled for a percentage of actors using the following command, it means the flag impacts 25% of the repositories. If a RPC does not have a repository, 25% of users are influenced instead. This is a side effect when the flags are pre-evaluated in Rails side. We could not pick the target actors!

/chatops run feature set gitaly-mep-mep 25 --actors

Screenshots or screen recordings

N/A. This MR does not introduce a change on the UI.

How to set up and validate locally

This section defines various testing scenarios. The main goal is to ensure Gitaly receives the correct list of relative feature flags in gRPC metadata. Unfortunately, Gitaly does not support logging feature flags by default. Therefore, we must add featureflag.UnaryInterceptor and featureflag.StreamInterceptor to gitaly server, and rebuild Gitaly code base:

diff --git a/internal/gitaly/server/server.go b/internal/gitaly/server/server.go
index e62b4b99c..180b31d70 100644
--- a/internal/gitaly/server/server.go
+++ b/internal/gitaly/server/server.go
@@ -3,6 +3,7 @@ package server
 import (
        "crypto/tls"
        "fmt"
+       "gitlab.com/gitlab-org/gitaly/v15/internal/middleware/featureflag"
        "time"
 
        grpcmw "github.com/grpc-ecosystem/go-grpc-middleware"
@@ -110,6 +111,7 @@ func New(
                sentryhandler.StreamLogHandler,
                cancelhandler.Stream, // Should be below LogHandler
                auth.StreamServerInterceptor(cfg.Auth),
+               featureflag.StreamInterceptor,
        }
        unaryServerInterceptors := []grpc.UnaryServerInterceptor{
                grpcmwtags.UnaryServerInterceptor(ctxTagOpts...),
@@ -126,6 +128,7 @@ func New(
                sentryhandler.UnaryLogHandler,
                cancelhandler.Unary, // Should be below LogHandler
                auth.UnaryServerInterceptor(cfg.Auth),
+               featureflag.UnaryInterceptor,
        }
        // Should be below auth handler to prevent v2 hmac tokens from timing out while queued
        for _, limitHandler := range limitHandlers {

Afterward, filter Gitaly logs to capture feature_flags field in each RPC with the following command:

gdk tail gitaly |  jq --stream --unbuffered --raw-input 'split("{")|.[1:]|join("{")|"{" + .|fromjson|{feature_flags, "grpc.request.fullMethod", "grpc.meta.client_name"}'

In the following scenarios, I'll use gitaly_mep_mep dummy feature flag for demonstration.

Flag is off globally

Click to expand

Feature.disable(:gitaly_mep_mep)

Clone via SSH: git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via HTTP: git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
  "grpc.meta.client_name": "gitlab-workhorse"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
  "grpc.meta.client_name": "gitlab-workhorse"
}

Visit the repository on UI http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
  "grpc.meta.client_name": "gitlab-web"
}

Flag is on globally

Click to expand

Feature.enable(:gitaly_mep_mep)

Clone via SSH: git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via HTTP: git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
  "grpc.meta.client_name": "gitlab-workhorse"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
  "grpc.meta.client_name": "gitlab-workhorse"
}

Visit the repository on UI http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
  "grpc.meta.client_name": "gitlab-web"
}

Flag is on for `gitlab-org/gitlab-shell`

Click to expand

Feature.remove(:gitaly_mep_mep)
Feature.enable(:gitaly_mep_mep, Project.find_by_full_path("gitlab-org/gitlab-shell").repository)

Clone via SSH: git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone another repo via SSH: git clone ssh://git@127.0.0.1:2222/gitlab-org/Flight.git

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via HTTP: git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
  "grpc.meta.client_name": "gitlab-workhorse"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
  "grpc.meta.client_name": "gitlab-workhorse"
}

Clone another repository via HTTP: git clone http://127.0.0.1:3000/gitlab-org/Flight.git

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
  "grpc.meta.client_name": "gitlab-workhorse"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
  "grpc.meta.client_name": "gitlab-workhorse"
}

Visit the repository on UI http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
  "grpc.meta.client_name": "gitlab-web"
}

Visit another repository on UI http://localhost:3000/gitlab-org/Flight/-/commits/main

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}

Flag is on for user Root

Click to expand

Feature.remove(:gitaly_mep_mep)
Feature.enable(:gitaly_mep_mep, User.find_by_username("root"))

Clone via SSH with root's public key git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via SSH with another use's public key git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via HTTP; this clone could not be associated with any user: git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
  "grpc.meta.client_name": "gitlab-workhorse"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
  "grpc.meta.client_name": "gitlab-workhorse"
}

Clone via HTTP with root basic auth: git clone http://root:password@127.0.0.1:3000/gitlab-org/gitlab-shell.git. Note that we could not determine the user from any public repository clone.

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
  "grpc.meta.client_name": "gitlab-workhorse"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
  "grpc.meta.client_name": "gitlab-workhorse"
}

Visit the repository on UI using user Root credentials http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
  "grpc.meta.client_name": "gitlab-web"
}

Visit the repository on UI using another user credentials http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
  "grpc.meta.client_name": "gitlab-web"
}

Some more tests with Personal Snippets, Project Snippets, and Designs

Flag is on for Group `gitlab-org`

Click to expand

Feature.remove(:gitaly_mep_mep)
Feature.enable(:gitaly_mep_mep, Group.find_by_path("gitlab-org"))

Clone via SSH git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via SSH git clone ssh://git@127.0.0.1:2222/gitlab-org/Flight.git

{
  "feature_flags": "true",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Clone via SSH git clone ssh://git@127.0.0.1:2222/root/tada.git

{
  "feature_flags": "false",
  "grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
  "grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}

Repeat similar scenarios with clone via HTTP
Visit the repository on UI http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main and http://localhost:3000/gitlab-org/Flight/-/commits/main

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}

Visit the repository on UI http://localhost:3000/root/tada/-/commits/main

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
  "grpc.meta.client_name": "gitlab-web"
}

Visit Project wiki http://localhost:3000/gitlab-org/Flight/-/wikis/home

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/TreeEntry",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}

Visit Project wiki of another org http://localhost:3000/flightjs/test-wiki-project/-/wikis/home

{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/TreeEntry",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
  "grpc.meta.client_name": "gitlab-web"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/LastCommitForPath",
  "grpc.meta.client_name": "gitlab-web"
}

Rollout to 25% of the repositories

Click to expand

It's time-consuming to test this style of rollout, I'll write a small script instead.

Feature.remove(:gitaly_mep_mep)
Feature.enable_percentage_of_actors(:gitaly_mep_mep, 25)

repositories = [
  Project.all.map(&:repository),
  Project.all.map { |p| p.wiki.repository },
  Group.all.map { |g| g.wiki.repository },
  Snippet.all.map(&:repository)
].flatten.compact.shuffle

repositories.each do |repository|
  begin
    repository.ls_files("main")
  rescue; end
end

Running the above scripts and analyzing the logs, all the logs have mep_mep feature flags. The number of mep_mep:true is 11/34. As the sample size is too small, I did not expect the enable count is equal to 25%. The result is close enough. In fact, it proves that the feature flag works for all repository types.

{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
  "grpc.meta.client_name": "gitlab-web",
  "grpc.request.glRepository": "wiki-8"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
  "grpc.meta.client_name": "gitlab-web",
  "grpc.request.glRepository": "project-14"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
  "grpc.meta.client_name": "gitlab-web",
  "grpc.request.glRepository": "project-4"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
  "grpc.meta.client_name": "gitlab-web",
  "grpc.request.glRepository": "group-25-wiki"
}
{
  "feature_flags": "mep_mep:true",
  "grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
  "grpc.meta.client_name": "gitlab-web",
  "grpc.request.glRepository": "wiki-6"
}
{
  "feature_flags": "mep_mep:false",
  "grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
  "grpc.meta.client_name": "gitlab-web",
  "grpc.request.glRepository": "snippet-4"
}
...
---
Total: 34
Enabled: 11

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Edited Nov 04, 2022 by Quang-Minh Nguyen

Fully support feature flag actors in Gitaly

What does this MR do and why?

List of supported actor types

Screenshots or screen recordings

How to set up and validate locally

Flag is off globally

Flag is on globally

Flag is on for gitlab-org/gitlab-shell

Flag is on for user Root

Flag is on for Group gitlab-org

Rollout to 25% of the repositories

MR acceptance checklist

Merge request reports

Flag is on for `gitlab-org/gitlab-shell`

Flag is on for Group `gitlab-org`