Syncing repositories being blocked by Secret Push Protection
We sync our canonical gitlab repo with our security repo. This sync goes both ways, and there have been problems with Secret Push Protection (SPP) when syncing both ways. To get around these problems, SPP has been temporarily disabled on each of those repos, but we need to investigate the root cause and come up with a solution.
On August 7th, problems were reported when syncing from security to canonical. SPP was temporarily disabled on ?canonical? (to be confirmed) to get past this problem.
August 7th details
Slack thread@dat.tang.gitlab: We are seeing some errors on the publish job https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/14925304#L97 https://gitlab.slack.com/archives/C0139MAV672/p1723021906515809. I will post the error message in the
2024-08-07 08:58:00.980077 F [security] ReleaseTools::RemoteRepository -- Failed to push -- {:remote=>:canonical, :ref=>"17-2-stable-ee", :output=>"remote: GitLab:
remote: PUSH BLOCKED: Secrets detected in code changes
remote:
remote: Secret push protection found the following secrets in commit: c953c1b094fe9c3041bd31c5215327da04fa3c74
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 08a4210bbbea1ed5594614ead5db6ac3443a45c6
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 49db6c4d6ac633e3d64f9c72a89c1af9f7c5d575
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: c8e32ed80fc316cdc09b542d98249887128bd572
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 344679c695bbeadb902df314182e6e2a17e6e58e
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 7f3afb042dc090e28554b24fff59612e6e1644fa
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 556512a36fa6f34f26dfc90dba5e078604ec1cda
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 9ad016dc8669af3f6c104627b1999aa4a65d25f9
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 2cd0aee38516cec2295e2354481b9f44dbde370e
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: 822d5acdf333c450b1f1e7f9ecc1ce926061a8cb
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: be9c1aa4c3b8f871a3dfc687279bc17d6b5f0503
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: Secret push protection found the following secrets in commit: d8533d727a1c036560df59282bf62ab561258a13
remote: -- spec/frontend/diffs/store/actions_spec.js:1244 | GitLab Personal Access Token
remote:
remote: To push your changes you must remove the identified secrets.
remote: For guidance, see https://gitlab.com/help/user/application_security/secret_detection/secret_push_protection/index.html#resolve-a-blocked-push
To gitlab.com:gitlab-org/gitlab.git
! [remote rejected] 17-2-stable-ee -> 17-2-stable-ee (pre-receive hook declined)
error: failed to push some refs to 'gitlab.com:gitlab-org/gitlab.git'
"}I see some changes 7365a54f, not sure if it's related
@djadmin: At first glance, it looked related to !161074 (merged) and a fix would simply be updating the spec to use sentence case.
However, looking at the job logs again (I don't have access to the project though), it seems related to Skip secret push protection feature. We have to skip the secret detection during the push because the changes which are to be released have dummy PATs in it. Alternatively, we can disable the Secret push protection for the time being if it's blocking the release.
Is it possible to append -o secret_push_protection.skip_all to the push command being used here?
git push -o secret_push_protection.skip_all
@dat.tang.gitlab:
the changes which are to be released have dummy PATs in it.
so sorry where are the PATs? the line https://gitlab.com/gitlab-org/security/gitlab/-/blob/17-2-stable-ee/spec/frontend/diffs/store/actions_spec.js#L1244 was added from March
Is it possible to append -o secret_push_protection.skip_all to the push command being used here?
I don't think we can add this to the automation. If it is the only solution, I have to do it manually then
@djadmin: We have a known limitation due to which the entire file is scanned, not just the diff. This is currently being worked upon in #469161 (closed).
Secret Push Protection scans all contents of modified files. This can cause a push to be unexpectedly blocked if a modified file contains a secret, even if the secret is not part of the diff.
https://docs.gitlab.com/ee/user/application_security/secret_detection/secret_push_protection/#push-blocked-unexpectedly I am certain that these are dummy PATs, and the file was touched with !156503 (merged).
Alternatively, you can disable the Secret push protection for the project (for the time being). I think that's a possibility because it must have been enabled manually for the project.
@dat.tang.gitlab: Thanks. I will temporary disable it and then reenable then.
it worked! Thanks for the help @dheeraj
On August 13th, the problem came back, but in reverse. There was an error when syncing from canonical to security. SPP was disabled on security to get past this.
August 13th details
@dat.tang.gitlab: I have a situation where the sync from canonical to security failed. Even though the error message was quite generic, I suspected it was the secret detection because we had a similar issue last week with it, thus I disabled the feature on the security repo . After that, I retried and the sync worked well.
The thing is, that feature was enabled on both canonical to security , so I don't know if there are any cases that a commit can slip through a push/MR but is blocked when syncing (which is also basically a push). Do you have any hint or sth we can check?
https://gitlab.slack.com/archives/C0139MAV672/p1723543198352259?thread_ts=1723542325.295399&cid=C0139MAV672
@rossfuhrman: I would recommend leaving the feature disabled on the security repo for now.
@DatTang this isn't currently blocking you or the releases, is it?
I don't know if there are any cases that a commit can slip through a push/MR but is blocked when syncing (which is also basically a push)
That could be what's going on, and there are a few ways I can think of that might be coming in to play here, but there could be others:
- A commit was pushed with the git push -o secret_push_protection.skip_all (docs). And then that flag doesn't apply to the sync
- We were scanning entire files for secrets, instead of just scanning the diff for commits. This led to detecting secrets that weren't part of any relevant commits. This is what we tried to fix with the change that got reverted yesterday (thanks again for your help with that).
- I'm not sure if this is a possibility based on timing, but it could be that a commit was made with a secret (hopefully a fake secret) before we had enabled the feature for the gitlab-org/gitlab repo, and then it got included in the sync. I think the feature was enabled for the gitlab-org/gitlab repo long enough ago that this isn't likely though.
@dat.tang.gitlab:
this isn't currently blocking you or the releases, is it? yes, if I leave it disabled
oh yes, there are many cases, why I didn't think about them
last week we had another issue, but it was another way around: the sync from security to canonical failed because it was disabled on security . Should we also disable the feature on canonical? I mean, I know it's a beta feature, so we want to test, and it's also a security check, so if you want, we can keep it enabled, but it will be more cumbersome for us when doing the release.
@rossfuhrman: It does seem that there are some problems with the feature when syncing. I also didn't realize that we are syncing both ways. We have a team meeting in 1 hour and I'd like to discuss this there.
For now, I'd like to leave it as is:
canonical: ON
security: OFF
But, we may want to turn canonical off after the meeting. We don't want to be blocking releases.
Also, if having it ON for canonical causes problems in the mean time, please feel free to turn it off.
I also think we'll create a new issue after the meeting for looking into this problem and syncing of repos.
@dat.tang.gitlab: thanks a lot Ross and the team. Let me know your decision after the meeting.
@rossfuhrman: I have created an Issue: #478813 (closed)
I'll be adding more details to the issue based on our Slack conversations and the SD team meeting.
In the team meeting, we also decided that secret push protection should be disabled on canonical so as to not cause problems with the release process. Is that something you have the rights to turn off @Dat Tang?
Config link
@dat.tang.gitlab: yes I can. Let me do it.
Done
Related feedback: #467408 (comment 2035619876)