Enable self-managed GitLab instances to verify gitlab.com tokens

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Enable self-managed GitLab instances to verify gitlab.com tokens through the Secret Detection Response Service (SDRS) by treating gitlab.com as an external service provider.

Problem Statement

Currently, when Secret Detection finds GitLab tokens in a self-managed instance:

  • Tokens are verified by querying the local database directly
  • This only works for tokens created on that specific instance
  • Self-managed customers cannot verify gitlab.com tokens that may have been leaked in their code
  • This creates a security blind spot where gitlab.com tokens go unverified

Many organizations use both self-managed GitLab and gitlab.com, making it critical to detect when gitlab.com tokens are accidentally committed to their self-managed instance.

Proposal

Extend the partner token verification system to handle gitlab.com tokens when running on self-managed instances:

  1. Token Detection Enhancement:

    • Detect gitlab.com-specific token patterns (e.g., glpat-, glptt-, gldt-)
    • Add logic to identify the token's origin (local vs gitlab.com)

  2. Routing Logic:

    • Local tokens: Continue using existing TokenLookupService
    • gitlab.com tokens: Route through SDRS as an external partner

  3. SDRS Integration:

    • Add gitlab.com as a partner provider in SDRS
    • Implement token verification using gitlab.com's public API
    • Use appropriate authentication (potentially requiring API token configuration)

Implementation Plan

Phase 1: Token Origin Detection

# In UpdateTokenStatusWorker
def gitlab_token_origin(token)
  # Check if token belongs to current instance
  return :local if TokenLookupService.exists?(token)
  
  # Check if it matches gitlab.com patterns
  return :gitlab_com if gitlab_com_token?(token)
  
  :unknown
end

def gitlab_com_token?(token)
  # Check for gitlab.com-specific patterns or prefixes
  token.match?(/^glpat-[a-zA-Z0-9_-]{20}$/) && !local_instance_token?(token)
end

Phase 2: SDRS GitLab.com Client

// partners/gitlab_com_client.go


type GitLabComClient struct {
    httpClient   *http.Client
    apiURL       string
    rateLimiter  *rate.Limiter
}

func (c *GitLabComClient) VerifyToken(ctx context.Context, token string) (*TokenStatus, error) {
    // Use gitlab.com API to verify token
    // Similar to other implementation but using GitLab API
}

Phase 3: Configuration for Self-Managed

  • Add admin setting to enable gitlab.com token verification
  • Allow configuration of gitlab.com API token for verification requests
  • Add rate limiting configuration

Technical Considerations

  1. Authentication:

    • May require self-managed instances to configure a gitlab.com API token
    • Consider using OAuth application for better security

  2. Rate Limiting:

    • Respect gitlab.com API rate limits
    • Implement caching for recently verified tokens

  3. Privacy:

    • Only send token value, not surrounding code context
    • Log verification attempts for audit

  4. Performance:

    • Async verification through existing SDRS infrastructure
    • No impact on Secret Detection pipeline performance

Success Metrics

  • Number of gitlab.com tokens detected in self-managed instances
  • Verification success rate
  • API rate limit compliance
  • Time to verify tokens
Edited by 🤖 GitLab Bot 🤖