### Problem to Solve Secret Push protection was initially built as an experimental feature for Dedicated customers, adn beta support was extended to .com customers in 17.1. We are taking on the work described below, to build a dedicated service that runs secret detection scans, which will make push protection more scalable, performant, and maintainable. This work is also required prior to building continuous secret detection scanning. ### Context In the [next phase](https://docs.gitlab.com/ee/architecture/blueprints/secret_detection/#phase-2---standalone-pre-receive-service) of Secret Push Protection, the goal is to have a dedicated service responsible for running Secret Detection scans on the given input blobs. This is done primarily from the scalability standpoint. Regex operations in the Secret Detection scan [consume](https://gitlab.com/gitlab-org/gitlab/-/issues/422574#note_1582015771 "Perform secret detection for highest risk content pre-receive") high resources so running scans within Rails or Gitaly instances would impact the resource availability for running other operations. Running scans in isolation provides greater control over resource allocation and scaling the service independently as needed. ### What is the plan ahead? ([**Epic Dashboard**](https://epic-dashboard-gitlab-org-tenant-scale-group-4aecf10d1d02154641.gitlab.io/epic_13792)) Phase 2 involves building components across ~"group::secret detection" and ~"group::gitaly" projects, which could take numerous milestones to reach the "definition of done". We've decided to rather deliver results iteratively by dividing Phase 2 into two **_iterations_** based on the technical implementation and engineering effort. The high-level plan for Phase 2 involves three things: 1. Build a standalone service responsible for running the Secret Detection scan on the input blobs. 2. Two iterations: 1. Move the Secret Detection scan operation from the existing Ruby Gem to a Standalone service for GitLab SaaS. 2. Introduce a provision for triggering Secret Push Protection from Gitaly(via binary plugin) instead of through Rails (current approach). #### Standalone Service: Implementation Plan * [x] [Technical Discovery for an RPC Service to run Secret Detection Scan](https://gitlab.com/gitlab-org/gitlab/-/issues/464711 "Build and deploy minimum viable Secret Detection RPC service in Runway") * [x] [Build and deploy minimum viable Secret Detection RPC service using Runway](https://gitlab.com/gitlab-org/gitlab/-/issues/464711 "Build and deploy minimum viable Secret Detection RPC service in Runway") * [x] [Make Secret Detection RPC service Production-ready](https://gitlab.com/gitlab-org/gitlab/-/issues/467531 "Make Secret Detection RPC service Production-ready") #### Iteration 1: Move the SD scan operation to a standalone service for GitLab SaaS In this iteration, the primary objective is to build a Secret Detection service responsible for running the Secret Detection scans. Due to certain [limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/462359#note_1915874628 "Technical Discovery: RPC Service to run Secret Detection scan") in Runway, we will continue using the current Ruby gem scanning approach for Self-managed instances for this iteration. Whereas for GitLab SaaS, we will offload the scanning to a standalone service by invoking it from Rails. To reuse the core implementation of Secret Scanning, we will have a single source code with two different distributions: 1. Wrap a Ruby gem around the source and use it in the Rails(replacing the current gem) 2. Wrap an RPC service around the source, deploy it using [Runway](https://gitlab.com/gitlab-com/gl-infra/platform/runway), and invoke the service from Rails for GitLab SaaS {width="1001" height="311"} The primary change in the workflow of Secret Push Protection would be the delegation of scanning responsibility from the [Secret Detection gem](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-secret_detection) to the RPC service for GitLab SaaS i.e., the [secrets push check](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/secrets_check.rb#L71) invokes the RPC service with an array of blobs to scan for secrets. Note that the access checks are still performed at the [Rails side](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/secrets_check.rb#L71). <details> <summary>See Workflow</summary> ```mermaid sequenceDiagram autonumber %% Phase 2: Iter 1 Gitaly->>+Rails: invokes `internal/allowed` Rails->>Rails: Perform access checks alt On access check failure Rails-->>Gitaly: Scanning Skipped end Rails->>Gitaly: Get blobs Gitaly->>Rails: Quarantined Blobs Note over Rails,SD Ruby Gem: For GitLab Self-Managed Rails->>SD Ruby Gem: Invoke RPC and forward quarantined blobs SD Ruby Gem->>SD Ruby Gem: Runs Secret Detection on input blobs SD Ruby Gem->>Rails: Result Note over Rails,SD RPC Service: For GitLab SaaS (GitLab.com & Dedicated) Rails->>SD RPC Service: Invoke RPC and forward quarantined blobs SD RPC Service->>SD RPC Service: Runs Secret Detection on input blobs SD RPC Service->>Rails: Result Rails->>Gitaly: Result ``` #### </details> #### Iteration 2: Stream blobs directly from Gitaly to the RPC service This iteration's main objective is introducing a provision for triggering Secret Push Protection from Gitaly through a binary placed within the Gitaly node. Until the previous iteration, there were multiple hops between Gitaly and Rails for running Pre-receive checks, particularly for Secret Push protection since a fairly large amount of Rails memory gets occupied for holding blobs to pass them to the Gem/RPC service for scanning. We can minimize this damage for GitLab SaaS environments by introducing a direct interaction between the RPC service and Gitaly through a standard interface (either [Custom pre-receive hook](https://docs.gitlab.com/ee/administration/server_hooks.html#create-global-server-hooks-for-all-repositories) or Gitaly's new [Plugin-based architecture](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143582 "Add a blueprint for Gitaly plugins")). This setup keeps the resource consumption in check and remains isolated within the Secret Detection service without hampering Gitaly or Rails resources. {width="752" height="235"} Gitaly's new [Plugin-based architecture](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/143582 "Add a blueprint for Gitaly plugins") would be the preferred interface for interacting b/w Gitaly and RPC service as it provides streamlined access to the git repository. However, at the time of creating this issue, it is not developed yet. Once the plugin interface is in place, the new git push event workflow will have the following steps: 1. On the git push event, Gitaly invokes the plugin giving it access to the repository and the new blobs in the quarantine. 2. The plugin relays the new blobs to the RPC service 3. The service performs the scan and responds to the plugin. 4. The plugin responds to Gitaly whether or not to accept the write and exits. _More details on iteration 3 will be added once there are updates on the development of Plugin architecture._ The expectations by the end of this iteration are: * We develop a binary that streams the blobs from Gitaly to the RPC service. RPC service scans the streamed blobs for secrets and returns results. * Gitaly invokes the binary on the pre-receive Git push event and we determine the status of the Secret push protection check based on the binary's invocation result. ### Tentative Timeline <table> <tr> <th> </th> <th>DRI</th> <th> Expected Release Milestone </th> <th>Status</th> </tr> <tr> <td>Build a standalone service responsible for running the Secret Detection scan on the input blobs</td> <td> @vbhat161 </td> <td>17.4</td> <td> ~"health::on track" </td> </tr> <tr> <td>Iteration 1 - Move the SD scan operation to a standalone service for GitLab SaaS</td> <td> @eurie </td> <td>17.5</td> <td> ~"health::on track" </td> </tr> <tr> <td>Iteration 2 - Stream blobs directly from Gitaly to the RPC service</td> <td> _TBD_ </td> <td> _TBD_ </td> <td> _TBD_ </td> </tr> </table>