Skip to content

Secret push protection status does not toggle via UI

Summary

We've had a couple reports (Slack: https://gitlab.slack.com/archives/C06NY8LDMT2/p1739547328729319, https://gitlab.slack.com/archives/C06NY8LDMT2/p1739210474732269) where the user attempts to enable Secret push protection via the UI, but after refreshing, the status goes back to disabled. See this video for an example: https://gitlab.slack.com/archives/C06NY8LDMT2/p1739560645186419?thread_ts=1739547328.729319&cid=C06NY8LDMT2

The user is able to enable the feature via REST API, tbd on GraphQL API.

This does not happen in all projects, so far it's unclear in what type of project it's happening.

Steps to reproduce

  1. Go to a project > Secure > Security configuration
  2. Attempt to toggle the status of Secret push protection from off to on
  3. Refresh the page
  4. Observe if the status is what you expect

Example Project

https://gitlab.com/gl-demo-ultimate-sissar/helm_chart/-/security/configuration

What is the current bug behavior?

The user attempts to enable Secret push protection via the UI, but after refreshing, the status goes back to disabled and SPP scans are not on.

https://gitlab.slack.com/archives/C06NY8LDMT2/p1739560645186419?thread_ts=1739547328.729319&cid=C06NY8LDMT2

What is the expected correct behavior?

The user attempts to enable Secret push protection via the UI, and after refreshing, the status stays enabled and SPP scans block secrets.

Relevant logs and/or screenshots

No error logs are generated

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Workaround

As a workaround, Secret push protection can be enabled via the REST API:

curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/security_settings?secret_push_protection_enabled=true"

Possible fixes

Need to check Project.where(secret_push_protection_enabled: nil).count and Project.where("secret_push_protection_enabled IS DISTINCT FROM pre_receive_secret_detection_enabled").count

I think it's possible that frontend toggle updates preReceiveSecretDetectionEnabled via GraphQL, but database and backend are using secret_push_protection_enabled. This mismatch means the toggle appears to switch in the UI, but the correct value isn’t saved or read back properly, causing it to reset after a refresh.

Possible fix is to merge !178811 (diffs)

I think my initial hunch was off, but I think I'm closer to the root cause:

After investigating a test project, I see that project.security_setting.secret_push_protection_enabled: nil . app/assets/javascripts/security_configuration/components/secret_push_protection_feature_card.vue:116 (which is the file that generates the UI toggle) checks that the column value isn't null before toggling the feature enablement status -- if it's null/nil, it doesn't toggle. I tested an existing different test project where I'd previously enabled/disabled SPP, and it shows p.security_setting.secret_push_protection_enabled => true, and I can confirm I'm able to toggle the status via the UI. I just created a new test project that's never had SPP enabled, and I was able to successfully toggle SPP and p.security_setting.secret_push_protection_enabled => false rather than nil. So I believe that this is happening to some projects that existed before the column rename that had never enabled SPP before. Potential maximum customer impact is ProjectSecuritySetting.where(secret_push_protection_enabled: nil).count => 10583626 😳

I think the reason for the secret_push_protection_enabled: nil is that the background migration to backfill the values of prsd_enabled into spp_enabled failed to complete for some reason -- when I check ProjectSecuritySetting.where("secret_push_protection_enabled IS false AND pre_receive_secret_detection_enabled IS false").count => 29422868 so we can see that the migration didn't just put nil for all projects that had never enabled SPP or had it explicitly disabled, it just failed to complete after 29422868 rows. Concerningly, ProjectSecuritySetting.where("secret_push_protection_enabled IS DISTINCT FROM pre_receive_secret_detection_enabled AND pre_receive_secret_detection_enabled IS true").count => 14946 so 14946 projects that had pre_receive_secret_detection_enabled => true now have secret_push_protection_enabled => nil 😞 So the report the other day in Slack where projects unexpectedly had their Secret push protection status disabled was likely also due to this. The migration I used was supposed to keep the data in sync so I'm not sure why it failed to complete 🤔 I can follow up with the DB team on this.

Anyway, to fix it:

  • Create a new migration to complete the backfill the nil values:
class BackfillNullSecretPushProtectionEnabled < ActiveRecord::Migration[6.0]
  disable_ddl_transaction!

  def up
      execute <<~SQL
        UPDATE project_security_settings
        SET secret_push_protection_enabled = pre_receive_secret_detection_enabled
        WHERE secret_push_protection_enabled IS NULL;
      SQL

      execute <<~SQL
        ALTER TABLE project_security_settings
        VALIDATE CONSTRAINT check_20a23efdb6; # this constraint was added in the column rename MR and it checks that spp_enabled is not null
      SQL
  end

  def down
    # No rollback needed as this just ensures consistency
  end
end

Regarding re-enablement, we could possibly ask someone with db write access to do something like:

UPDATE project_security_setting
SET secret_push_protection_enabled = pre_receive_secret_detection_enabled
WHERE pre_receive_secret_detection_enabled IS true AND secret_push_protection_enabled IS NULL;

which could be faster than the previously proposed solution of doing a backfill migration. This could solve the immediate concern of the 15k projects that should have SPP enabled. Then we could follow up with the backfill migration for the other ~10M projects where SPP is expected to be false, but the column is currently nil (preventing the UI toggle from working)

Edited by Serena Fang