Partner Token Verification - Feature Flag and Rollout Plan

Summary

This issue is to roll out partner token verification on production, currently behind the secret_detection_partner_token_verification feature flag.

Owners

  • Slack channel: #g_ast-secret-detection
  • DRI: @atiwari71

Expectations

What are we expecting to happen?

Partner APIs (AWS, GCP, Postman) will verify if detected tokens are valid/active, providing users with accurate token validity status.

What can go wrong and how would we detect it?

  • Partner API errors: High error rates (>10%) or network failures
  • Performance degradation: P95 latency >5s causing timeouts
  • Rate limit exhaustion: Hitting partner API limits

Detection:

  • Dashboard
  • Alerts: SecretDetectionPartnerAPIHighErrorRate, SecretDetectionPartnerAPIHighLatency

Rollout Steps

Rollout on non-production environments

  • 19 Nov: Enable globally on non-production environments
/chatops run feature set secret_detection_partner_token_verification true --dev --pre --staging --staging-ref
  • Verify feature works in staging-canary
  • Monitor error rates and latency on dashboard

Specific rollout on production

Execute in #production, cross-post to #g_ast-secret-detection:

  • 20 Nov: Enable for GitLab.org
/chatops run feature set --group=gitlab-org,gitlab-com secret_detection_partner_token_verification true
  • Verify feature works for GitLab.org projects
  • Monitor dashboard for 24 hours

Global rollout on production

  • 1 Dec: Enable globally on GitLab.com
/chatops run feature set secret_detection_partner_token_verification true
  • Monitor for at least 24 hours

Release the feature

  • 8 Dec: Create MR to enable feature flag by default

Remove FF

  • 12 Jan: Create MR to remove feature flag
    • Remove all code references
    • Remove YAML definition
  • After merge and deploy, delete flag:
/chatops run feature delete secret_detection_partner_token_verification --dev --pre --staging --staging-ref --production
  • Close this issue

Rollback Steps

  • Disable on production:
/chatops run feature set secret_detection_partner_token_verification false
  • Disable on non-production:
/chatops run feature set secret_detection_partner_token_verification false --dev --pre --staging --staging-ref
Edited by Aditya Tiwari