Technical Discovery: RPC Service to run Secret Detection scan
Purpose of the issue
This discovery issue combines multiple related tasks under a single issue. This issue outlines all the necessary elements for building a standalone Go service for running Secret Detection scans.
Questions to address
Here are some of the questions this issue should address to mark as complete. Some of the questions are not relevant to Iteration-1 but are for other iterations, however, it is important to identify any gaps at the earlier stage so that we can pivot towards the feasible direction.
-
Programming Language: Which programming language is best for building the RPC service - Ruby or Go? Considering Performance, Tooling reuse, and Team's convenience in mind. | Due to some limitations, we will continue with Ruby for now -
Infrastructure: -
How is the service deployed with GitLab CI/CD integration for: -
GitLab SaaS | We will deploy RPC service using Runway -
GitLab Self-managed instances | We will continue using the current Ruby gem approach temporarily due to certain reasons.
-
-
What are the Service Level Indicators(SLIs) to determine the service's performance? -
What are the SLOs defined for the service?
-
-
Platform tooling: How will the service integrate with: -
GitLab Logging infrastructure -
Sentry for Error Monitoring -
Prometheus for System Monitoring -
Capturing Metrics for Observability -
Pager alerts on failures
-
-
RPC Service: -
How do we ensure SD scan memory consumption is within limits? Do we need to adopt any approach similar to scan within the subprocess approach followed in the SD Ruby gem? -
What is the RPC request and response specification for invoking a Secret Detection scan by the external services? -
Do we use any standard cookie-cutter project like Labkit? and, which developer dependencies (linting/testing/etc) do we plan on using for the project? -
What is the release process followed for the Service and Ruby gem? How does it align with GitLab milestone releases - do we bump for both major & minor releases or just major milestone releases similar to Secure analyzers? (Keep Self-managed instances in mind when deciding) -
Where does the source code reside? Is it under gitlab-org/security-products
group or should a dedicated group be created for Secrets Detection undergitlab-org/security-products
?
-
-
Streaming Binary client (we may opt out of answering questions for this section at this moment and answer them before proceeding with Iteration-3 since the implementation isn't finalized on the target binary platform): -
How is the binary packaged and distributed with Continuous deployment using Omnibus? -
How is the binary deployed with Continuous deployment in the Gitaly node? -
What is the release process followed for the binary? Does it align with GitLab milestones or Gitaly releases? -
Where does the source code for binary reside?
-
Definition of done
-
Conduct a tradeoff between Go and Ruby and decide the programming language of choice for the RPC Service -
Add an ADR entry in the Platform-wide Secret Detection section outlining the proposed approach for building a standalone Go service. This should include all the decisions that are answered in the above questions. | Add Standalone Secret Detection Service in the ... (!154891 - merged) -
Answer most of the questions (if not all) under the Questions to Address section.
Reference links (to be elaborated):
- Standalone Service concept: https://docs.gitlab.com/ee/architecture/blueprints/gitlab_ml_experiments
- Service Deployment & Docs: https://gitlab.com/gitlab-com/gl-infra/platform/runway (Docs)
- Platform Toolkit for writing Go service: https://gitlab.com/gitlab-org/labkit
Edited by Vishwa Bhat