Add Secret Detection Validity Checks runbooks

  * [Monitoring](secret-detection-svc-monitoring)

  * _Troubleshooting (TBA, once we complete [#494910](https://gitlab.com/gitlab-org/gitlab/-/issues/494910) and [#499249](https://gitlab.com/gitlab-org/gitlab/-/issues/499249))_

* Secret Validity Check

  * [General FAQs](secret-detection-validity-check-faqs)

  * [Monitoring](secret-detection-validity-check-monitoring)

  * External Token Verification

    * [Troubleshooting](https://runbooks.gitlab.com/secret-detection/troubleshooting)

    * [Troubleshooting](secret-detection-validity-check-troubleshooting)

---

title: "Secret Detection Validity Checks: General FAQs"

---

### When to use this runbook?

Use this runbook to understand the validity checks feature, answer common questions, and

troubleshoot basic configuration issues. For detailed troubleshooting, see the

[troubleshooting guide](secret-detection-validity-check-troubleshooting).

### What is validity checks?

Validity checks verifies token status by calling partner APIs (AWS, GCP, Postman) and checking

GitLab token database. Status: **Active**, **Inactive**, or **Unknown**.

## Feature availability

| Offering | Tier | Status |

|----------|------|--------|

| GitLab.com | Ultimate | GA |

| GitLab Dedicated | Ultimate | GA |

| GitLab Self-Managed | Ultimate | GA |

## How it works

GitLab tokens are checked immediately through database query during report ingestion.

Partner tokens are verified asynchronously through Sidekiq worker with rate limiting.

Tokens show "Checking..." until verification completes (typically less than one minute).

## Enabling the feature

1. Go to **Secure > Security configuration > Secret detection**.

2. Turn on **Secret validity checks**.

## Status meanings

| Status | Meaning |

|--------|---------|

| Active | Token is valid |

| Inactive | Token revoked or expired |

| Unknown | Verification failed |

## Manual refresh

Select **Verify token status** on the vulnerability details page to refresh status on-demand.

## Token privacy

Token strings are sent to partner APIs for validation:

- AWS, GCP, Postman: Tokens sent to partner endpoints

- GitLab: Tokens never leave GitLab (database only)

## Disable the feature

Project-level: Users with at least the Maintainer role can turn on the setting in **Security configuration**.

## Where to find status

- Vulnerability details page: Under the **Validity** badge

- Security dashboard: Filter by token status

- Vulnerability API: Included in response

## Performance impact

Validity checks run asynchronously and have minimal performance impact. GitLab tokens take less than 100 ms to verify.

Partner tokens take 500 ms to 5 seconds to verify.

## Token check frequency

Tokens are checked once per scan automatically. You can also check tokens manually on-demand through the UI.

## Unknown token status

A token might show as `Unknown` for several reasons:

- Partner API is down

- Rate limited

- Invalid token format

- Unexpected response

For more information, see [troubleshooting secret detection validity checks](secret-detection-validity-check-troubleshooting).

## Report issues

File issues in the `#g_ast-secret-detection` Slack channel or

[GitLab tracker](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=group%3A%3Asecret%20detection).

## Related documentation

- [Monitoring guide](secret-detection-validity-check-monitoring.md)

- [Troubleshooting](secret-detection-validity-check-troubleshooting.md)

- [Architecture design](/handbook/engineering/architecture/design-documents/secret_detection_validity_checks/)

---

title: "Secret Detection Validity Checks Monitoring"

---

### When to use this runbook?

Use this runbook to monitor the validity checks feature health, performance, and usage.

For more information, see the [validity checks dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2).

### What to monitor?

Validity checks spans these components:

- Sidekiq workers (background job processing)

- Partner APIs (AWS STS, GCP OAuth2, Postman)

- Rails (finding ingestion and status updates)

- Redis (rate limiter, queue depth)

- PostgreSQL (`secret_detection_token_statuses` and `security_finding_token_statuses` tables)

## Key metrics

### Request rate

| Metric | Source | Meaning |

|--------|--------|---------|

| Requests per sec by partner | `validity_check_partner_api_requests_total` | API call volume per partner |

| Success rate by partner     | `validity_check_partner_api_requests_total{status="success"}` | Percentage of successful verifications |

| Failure rate by partner     | `validity_check_partner_api_requests_total{status="failure"}` | Percentage of failed verifications |

Dashboard: **Requests per Second by Partner**, **Success Rate by Partner**, and **Failure rate by partner**

### Latency

| Metric | Target | Alert |

|--------|--------|-------|

| API Response P95 | < 5s | > 5s for 5+ min |

Measured using `validity_check_partner_api_duration_seconds_bucket` (95th percentile).

Dashboard: **API Response Time (P95)**

### Errors

| Type | Metric | Meaning |

|------|--------|---------|

| Overall error rate | `validity_check_partner_api_requests_total{status="failure"}` | % of all requests failing |

| Network errors | `validity_check_network_errors_total` by error class | Connectivity issues |

| Error breakdown | `validity_check_partner_api_requests_total` by error type | Failure categorization |

Possible error classes: `Timeout`, `ConnectionRefused`, `HTTPError`

Dashboard: **Errors by Type**, **Network Errors by Partner**, and **Error breakdown**

### Rate limiting

| Metric | Alert |

|--------|-------|

| Rate limit hits/sec | > 0.1 req/sec sustained |

Measured with `validity_check_rate_limit_hits_total` (requests rejected due to limits).

Location: **Rate Limit Hits**, **Rate Limits by Partner**

## Common alerts

### `SecretDetectionPartnerAPIHighErrorRate`

- Severity: S3

- Threshold: > 10% for 5+ min

- Check: **Current Error Rate** gauge, **Success Rate by Partner** chart

- Action: Identify affected partner(s). Check partner status pages. For more information, see the

[troubleshooting](secret-detection-validity-check-troubleshooting.md#partner-api-high-error-rate) section.

### `SecretDetectionPartnerAPIHighLatency`

- Severity: S3

- Threshold: P95 > 5s for 5+ min

- Check: **API Response Time (P95)** chart

- Action: Check if systemic (all partners) or partner-specific. For more information, see the [troubleshooting](secret-detection-validity-check-troubleshooting.md#partner-api-high-latency) section.

### `SecretDetectionPartnerAPIRateLimitHit`

- Severity: S4

- Threshold: > 0.1 req/sec sustained

- Check: **Rate Limit Hits** stat, **Rate Limits by Partner** chart

- Action: For more information, see the [troubleshooting](secret-detection-validity-check-troubleshooting.md#rate-limits-hit-frequently) section.

### `SecretDetectionPartnerAPINetworkErrors`

- Severity: S3

- Threshold: > 0.5 errors/sec, 5+ min

- Check: **Network Errors by Partner** chart

- Action: Identify error type (`Timeout`, `ConnectionRefused`, `HTTPError`). For more information, see the [troubleshooting](secret-detection-validity-check-troubleshooting.md#network-errors-to-partner-apis) section.

## Dashboards

| Dashboard | Purpose |

|-----------|---------|

| [Validity checks](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2) | Request rate, latency, errors, rate limits |

| [Sidekiq workers](https://dashboards.gitlab.net/dashboards?query=sidekiq) | Job processing, retries |

| [PostgreSQL tables](https://dashboards.gitlab.net/dashboards/f/postgresql/postgresql) | `finding_token_status` growth |

## Panel descriptions

### Current Error Rate

Real-time error rate (%) across all partners. Combines success and failure counts. Alert fires if the rate exceeds 10%.

### API Response Time (P95)

95th percentile latency for each partner API call. Spikes indicate partner slowdowns. Alert fires if the latency exceeds 5 seconds for 5 or more minutes.

### Rate Limit Hits

Current rate at which partner limits are being hit (req/sec). Zero is healthy. Alert fires if the rate exceeds 0.1 req/sec.

### Requests per Second by Partner

API call volume to each partner. Use to see usage patterns.

### Success Rate by Partner

Percentage of successful verifications per partner. Should stay above 95%.

### Errors by Type

Failure reasons (network, rate limit, response parsing). Stack chart shows composition.

### Network Errors by Partner

Detailed breakdown: `Timeout`, `ConnectionRefused`, `HTTPError`. Diagnose connectivity issues.

### Rate Limits by Partner

Shows which partner limits you're hitting. Shows `limit_type` (per-second threshold).

## Performance baselines

No published SLOs exist. Use dashboard P95 latency as baseline. Expect fluctuation based on partner API health.

## Escalation

- Team: Secret Detection (@gitlab-org/secure/secret-detection)

- Slack: `#g_ast-secret-detection`

- On-call: Check `#production` for SRE availability

---

title: "Secret Detection Validity Checks Troubleshooting"

---

### When to use this runbook?

Use this runbook for troubleshooting production issues related to the validity checks feature.

## Monitoring

[Validity checks monitoring](secret-detection-validity-check-monitoring) is the preferred

dashboard for identifying reliability issues.

[Sidekiq dashboard](https://dashboards.gitlab.net/d/sidekiq-workers) can supplement, follow [runbook](https://gitlab.com/gitlab-com/runbooks/-/tree/master/docs/sidekiq).

## Partner API high error rate

### Symptom

Error rate exceeds 10% on [dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2)

(alert: `SecretDetectionPartnerAPIHighErrorRate`).

### Investigation

1. Identify affected partner(s) in dashboard breakdown.

1. Check error type:

   - `network_error`: Connectivity issues

   - `rate_limit`: Rate limit exceeded

   - `response_error`: Invalid/unparseable response

1. Review partner status pages:

   - [AWS](https://status.aws.amazon.com)

   - [GCP](https://status.cloud.google.com)

   - [Postman](https://status.postman.com)

1. Check recent deployments:

   ```shell

   git log --oneline -10 ee/lib/security/secret_detection/partner_tokens/

### Resolution

- If partner has known incident (< 1 hour): Wait for recovery

- If network error (multiple partners): Coordinate with SRE in #production

- If response error: Likely API contract change - file issue in `#g_ast-secret-detection`

- If persistent (> 6 hours): [Disable partner](#disable-partners)

## Partner API high latency

### Symptom

P95 latency exceeds 5 seconds on the [dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2) (alert: `SecretDetectionPartnerAPIHighLatency`).

### Investigation

1. Check if systemic (all partners) or partner-specific.

1. Review partner status pages for degraded performance.

1. Test partner API from teleport Rails console:

   ```ruby

   uri = URI('https://sts.amazonaws.com/')

   start = Time.now

   Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|

     http.request(Net::HTTP::Get.new(uri.path))

   end

   puts "Latency: #{Time.now - start}s"

   ```

### Resolution

- P95 less than 10 seconds and improving: Monitor—issue resolves itself

- P95 exceeds 10 seconds sustained (more than 24 hours): [Disable partner](#disable-partners)

## Rate limits hit frequently

### Symptom

Exceeds 0.1 req/sec sustained for 5 or more minutes on the [dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2) (alert: `SecretDetectionPartnerAPIRateLimitHit`).

### Investigation

1. Identify partner(s) hitting limits.

1. Check current limits in `ee/lib/gitlab/application_rate_limiter.rb`:

   ```ruby

   partner_aws_api: { threshold: -> { 400 }, interval: 1.second }

   partner_gcp_api: { threshold: -> { 500 }, interval: 1.second }

   partner_postman_api: { threshold: -> { 4 }, interval: 1.second }

   ```

1. Check Sidekiq queue depth using teleport:

   ```ruby

   Sidekiq::Queue.new('security_secret_detection_partner_token_verification').size

   ```

1. Check for burst patterns: Large pipeline or multiple projects scanning.

### Resolution

1. Queue less than 1k jobs: Normal, will clear within 1 hour

1. Queue 1k–10k jobs: Monitor queue size—rate limiter auto-throttles processing

1. Queue exceeds 50k jobs: Temporarily [disable partner](#disable-partners)

1. Persistent throttling (more than 24h): Update rate limits or disable partner

## Network errors to partner APIs

### Symptom

Exceeds 0.5 errors/sec for 5 or more minutes on the [dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2) (alert: `SecretDetectionPartnerAPINetworkErrors`).

### Investigation

1. Check which partner(s) affected.

1. Verify connectivity from Rails console using teleport:

   ```ruby

   # Test AWS

   uri = URI('https://sts.amazonaws.com/')

   begin

     Net::HTTP.get_response(uri)

     puts "✓ AWS reachable"

   rescue => e

     puts "✗ AWS unreachable: #{e.message}"

   end

   # Test GCP

   uri = URI('https://www.googleapis.com/oauth2/v1/tokeninfo')

   begin

     Net::HTTP.get_response(uri)

     puts "✓ GCP reachable"

   rescue => e

     puts "✗ GCP unreachable: #{e.message}"

   end

   ```

1. Check DNS:

   ```shell

   dig sts.amazonaws.com

   ```

1. Check for firewall changes in `#infrastructure`.

1. Check logs for SSL errors using data view `pubsub-sidekiq-inf-gprd*`:

   ```plaintext

   json.class:PartnerTokenVerificationWorker AND "SSL"

   ```

### Resolution

- Single partner affected: Likely partner-side disabled partner, monitor status page.

- Multiple partners affected: Likely GitLab network issue, coordinate with SRE.

- SSL/TLS errors: Check certificate validity, might need CA bundle update.

## Disable partners

### Emergency disable (GitLab.com)

Edit `ee/lib/security/secret_detection/partner_tokens/registry.rb`:

```ruby

'AWS' => {

  client_class: ::Security::SecretDetection::PartnerTokens::AwsClient,

  rate_limit_key: :partner_aws_api,

  enabled: false  # ← Set to false

}

```

### Re-enable after incident

1. Verify issue is resolved: Partner status is operational, network OK, error rate down.

1. Make code change to re-enable partner.

1. Monitor the [dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2)

for 10 minutes: Error rate less than 2%, queue depth less than 1k, P95 less than 5 seconds.

## Manual token verification

To verify a token without waiting for the background job, use the Rails console (teleport):

```ruby

finding = Vulnerabilities::Finding.find(FINDING_ID)

token_type = finding.identifiers.find { |i|

  i['external_type'] == 'gitleaks_rule_id'

}&.dig('external_id')

partner_config = Security::SecretDetection::PartnerTokens::Registry.partner_for(token_type)

client = partner_config[:client_class].new

result = client.verify_token(finding.metadata['raw_source_code_extract'])

puts "Valid: #{result.valid}"

puts "Metadata: #{result.metadata}"

```

## Escalation

- Team: Secret Detection (@gitlab-org/secure/secret-detection)

- Slack : `#g_ast-secret-detection`

- On-call: Check `#production` for SRE availability

- Quick checklist:

  - [ ] Issue confirmed on the [dashboard](https://dashboards.gitlab.net/d/secret-detection-sd-partner-token-verify/93b6ec2)

  - [ ] Checked partner status pages

  - [ ] Reviewed recent code deployments

  - [ ] Checked logs for errors

  - [ ] Tried turning off partner (for emergency situations)