Skip to content

Performance review for gitlab:doctor:secrets

Summary

Running doctore:secrets

This MR was merged in version 18.1 to work on "Performance improvements for gitlab:doctor:secrets" but the customer's environment which is already on 18.2.1 still takes hours to complete.

More context in the internal link to the ticket.

Steps to reproduce

  • Reproduced on customer environment.

What is the current bug behavior?

Truncated results show Ci::Build takes more than 4 hours:

I, [2025-08-08T08:04:07.252972 #1829025]  INFO -- : Checking encrypted values in the database
...
I, [2025-08-08T08:22:27.843349 #1829025]  INFO -- : - Group failures: 0
I, [2025-08-08T08:22:27.868914 #1829025]  INFO -- : - Ci::Runner failures: 0
I, [2025-08-08T12:47:04.244186 #1829025]  INFO -- : - Ci::Build failures: 0
I, [2025-08-08T12:47:04.244376 #1829025]  INFO -- : Total: 0 row(s) affected
I, [2025-08-08T12:47:04.244407 #1829025]  INFO -- : Done!

Checking the number of non-NULL token_encrypted values shows there are very few. If these changes took effect, the task should not go through the NULL tokens.

Total CI builds: 1699746
Builds with NULL token_encrypted: 1699674
Builds with non-NULL token_encrypted: 71
Percentage with NULL tokens: 100.0%
Percentage with non-NULL tokens: 0.0%

What is the expected correct behavior?

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

 (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`)  (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 

(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`)

(For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)

(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖