Integrate Secret Detection service with Sentry for Error Tracking

Context

Currently, the errors raised in the Secret Detection service are tracked via service logs only during the occasion of service failure (via monitoring alerts). However, we need to track recurrent application errors raised due to logical issues in the code.

GitLab uses Sentry tool for application error monitoring. We could integrate it with the Secret Detection Service to address the above-mentioned problem.

Proposed Solution

The proposal suggests a two-phase solution based on the effort involved:

  1. Iteration 1: Integrate GitLab Integrated Error Tracking for the gRPC service.
  2. Iteration 2: Integrate Sentry Error Tracking for both gRPC and Secret Detection gem

Iteration 1: Integrate GitLab Integrated Error Tracking for the gRPC service

GitLab Integrated Error Tracking uses GitLab UI (Monitor > Error Tracking section) to display captured errors while it internally uses Sentry SDK to perform error tracking. Integrated Error Tracking covers only the basic use cases of error tracking like the stack-trace, release version, no of occurrences and first & last seen (see sample for reference).

Pulling the implementation steps from this comment:

  • Follow these steps in the service repository to generate Sentry DSN value. Since the Sentry DSN value is sensitive, we should store it securely in the Hashicorp vault and read it in the runtime, as we did for AUTH_TOKEN. | Vault location for SD_TRACKING_DSN ✅
  • Create a new variable say, SENTRY_DSN, in both the staging and production environments.
  • Install the Sentry ruby gem in the service code and set SENTRY_DSN value when initializing Sentry SDK in the code.
  • Capture errors in Sentry for unknown grpc errors and the errors propagated by the core logic to grpc layer.
  • Verify if the errors are showing up. | Sample ✅

Merge Request: gitlab-org/security-products/secret-detection/secret-detection-service!58 (merged)

Iteration 2: Integrate Sentry Error Tracking for both gRPC and Secret Detection gem

SDS

  • Create a new Sentry project. For each GitLab project that you want to integrate, create a new Sentry project. (Seek help from Observability team since we don't have permissions)
  • Get the DSN for the new project in Sentry
  • Update the secret in Vault with the Sentry DSN
  • Redeploy the SDS to read the new vault value
  • Verify if the errors are showing up

Gem

  • Add the env var to monolith's runtime environment so the gem can access it
  • Verify if the errors are showing up
Edited by Ethan Urie