Use the SDS hosted on runway for GitLab.com

What does this MR do and why?

This MR implements the ability to route all GitLab.com secret push protection checks to the Secret Detection Service (SDS) hosted on Runway instead of using the local gem-based scanner, helping to reduce load on the GitLab.com monolith.

What it does:

• Adds a new feature flag spp_use_sds_for_gitlab_com to control SDS usage for GitLab.com
• Modifies the secret push protection flow to prioritize SDS hosted on Runway when the feature flag is enabled
• Implements fallback logic to use the gem-based scanner when SDS is unavailable or returns errors
• Removes the previous threading SDS test
• Adds proper error handling and logging when SDS requests fail

References

Route all GitLab.com secret checks to SDS (#577770) • Craig Smith • 18.6

Screenshots or screen recordings

Before	After

How to set up and validate locally

To test this change locally, you'll need to set up the Secret Detection Service (SDS) and verify that secret push protection requests are routed through it instead of the local gem.

Prerequisites

1. Set up SDS locally:

   • Clone the Secret Detection Service repository
   • Start the service with Docker:

     make grpc_docker_build && make grpc_docker_serve

   • The service will be available on localhost:8080

2. Prepare a test project:

   • Create a test project with Secret Push Protection enabled
   • Ensure the project contains a README.md file for testing

Local Development Setup

To force your local GitLab instance to use SDS instead of the gem-based scanner, apply this temporary patch:

diff --git a/ee/lib/gitlab/checks/secret_push_protection/secret_detection_service_client.rb b/ee/lib/gitlab/checks/secret_push_protection/secret_detection_service_client.rb
index 456985f32a7c..952bf8862de7 100644
--- a/ee/lib/gitlab/checks/secret_push_protection/secret_detection_service_client.rb
+++ b/ee/lib/gitlab/checks/secret_push_protection/secret_detection_service_client.rb
@@ -24,6 +24,7 @@ def initialize(project:)

        # Determines if SDS should be used (feature flags + instance checks).
        def use_secret_detection_service?
+          return true
          return should_use_sds unless should_use_sds.nil?

          sds_ff_enabled = Feature.enabled?(:use_secret_detection_service, project)
@@ -47,7 +48,8 @@ def setup_sds_client
          begin
            @sds_client = ::Gitlab::SecretDetection::GRPC::Client.new(
              sds_host,
-              secure: sds_auth_token.present?,
+              # secure: sds_auth_token.present?,
+              secure: false,
              logger: secret_detection_logger
            )
          rescue StandardError => e

This patch:

• Forces use_secret_detection_service? to return true
• Disables TLS for local development (since local SDS runs without TLS)

Testing Secret Detection

1. Trigger a secret detection scan in the test project:

   echo "glpat-00000000000000000000" >> README.md
   git add --all
   git commit -m "Add test secret"
   git push origin main

2. Verify the behavior:

   • The push should be blocked by secret push protection
   • Check the GitLab logs to confirm the request was sent to SDS rather than processed by the local gem
   • Look for log entries indicating SDS communication in the secret detection logs

3. Test fallback behavior:

   • Stop the SDS service (docker stop the SDS container)
   • Attempt the same push - it should still be blocked, but now using the gem-based fallback
   • Check logs for the fallback warning message: "SDS returned a nil response. Falling back to SDS Gem"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.