Skip to content

Integrate partner verification

Aditya Tiwarirequested to merge
567734-integrate-partner-verification into master

What does this MR do and why?

This is a 2 part mr for the Integrate Partner Verification into UpdateToken... (#567734), second part can be found here !207069.

This MR implements the foundational framework for partner token verification using direct API calls from the GitLab monolith via Sidekiq workers, replacing the previous separate SDRS service approach.

Note: Currently, this feature is behind the development feature flag and this code won't change anything for customers until the FF is enabled and the whole workflow is built as the part of [GA] Validity Checks (&16890).

Problem: Currently, partner tokens (AWS, GCP, Postman) detected by secret detection are not verified for validity, potentially leading to false positives and unnecessary alerts.

Solution: This MR introduces:

  • A base service class for partner token verification that can be extended for different finding types
  • Integration with the existing secret detection pipeline to automatically trigger partner verification
  • Asynchronous processing via Sidekiq workers to handle API rate limits
  • Feature flag for gradual rollout

Understanding the Current System

UpdateTokenStatusService handles 2 categories of tokens:

  1. GitLab tokens (and unknown types)
  2. Partner tokens (AWS, GCP, Postman)

And processes them through 2 workflows:

  1. Pipeline workflow - Async, multiple tokens at once
  2. Finding workflow - Sync from UI, one token at a time

How Are Tokens Actually Processed?

GitLab Tokens

(Yes, this is complex - we're refactoring it in #562175)

Partner Tokens (New)

  • PartnerTokenService filters for partner tokens only
  • Checks one token at a time (by partner)
  • Updates individually (no bulk)

Why one at a time?

  • Partner APIs only support single-token validation
  • Each partner has different response times

Are We Doing Duplicate Upserts?

No. Let me show you what's actually happening:

Upsert 1 (GitLab tokens):

process_findings_batch(batch, :vulnerability)  # Only processes GitLab tokens
  • Filters to GitLab tokens only
  • Bulk update via upsert_all

Upsert 2 (Partner tokens):

PartnerTokenService.process_finding_async(batch)  # Only processes partner tokens
  • Filters to partner tokens only
  • Individual updates

Key point: These handle completely different sets of tokens. No overlap.

Think of it like:

Batch of 10 tokens:
├─ 7 GitLab tokens → Upsert #1 (bulk)
└─ 3 Partner tokens → Upsert #2 (individual)

Zero duplication.

Are We Doubling Database Write Traffic?

Let's look at a real example:

Pipeline detects 10 tokens:

  • 7 GitLab PATs
  • 2 AWS keys
  • 1 Postman token

Before this MR:

Database writes: 1
  └─ Bulk write for 7 GitLab tokens
Unverified: 3 partner tokens (ignored)

After this MR:

Database writes: 4
  ├─ 1 bulk write (7 GitLab tokens)
  ├─ 1 individual write (AWS #1)
  ├─ 1 individual write (AWS #2)
  └─ 1 individual write (Postman)
Unverified: 0

Are we okay with this?

✅ Yes, because:

  • We're not duplicating writes for the same tokens
  • GitLab tokens stay efficient (bulk)
  • Partner tokens weren't being processed at all before
  • This is new functionality, not redundant work
  • Feature flag controls rollout

What Changes for Users?

Before (FF disabled):

User commits:
├─ 1 GitLab PAT → ✅ Status shown (active/inactive)
└─ 1 AWS key → ❌ Detected but no status shown

After (FF enabled):

User commits:
├─ 1 GitLab PAT → ✅ Status shown (active/inactive)
└─ 1 AWS key → ✅ Status shown (active/inactive)

Simple: We go from verifying only GitLab tokens to verifying both GitLab and partner tokens.

Here is a video demo of Gitlab token workflow, partner token will look same for users - Video demo

How to set up and validate locally

  1. Enable the feature flag:

    Feature.enable(:secret_detection_partner_token_verification)

  2. Create a project with secret detection enabled

  3. Push code containing a partner token (AWS, GCP, or Postman) to default branch.

  4. Check that token status is saved in the appropriate table

  5. Check the vulnerability report to see the status of these tokens.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related issues

Closes Integrate Partner Verification into UpdateToken... (#567734)

Screenshots or screen recordings

N/A - Backend changes only

Implementation Details

Class Structure

PartnerTokenServiceBase (new base class)
├── process_finding_async - Enqueues Sidekiq workers for async processing
├── process_partner_finding - Performs synchronous partner API verification
├── save_result - Persists verification results
└── partner_token? - Checks if token type requires partner verification

Vulnerabilities::PartnerTokenService (refactored)
└── Inherits from PartnerTokenServiceBase
    └── Implements abstract methods for vulnerability findings

Documentation

  • Documentation added/updated (if needed)
  • Feature flag documented in YAML
Edited by Aditya Tiwari

Merge request reports