Bug with secrets in multiple commits where the earlier commit shows blob_id instead of commit_id
Summary
Bug with secrets in multiple commits where the earlier commit shows blob_id instead of commit_id, possibly related to GetTreeEntries
Vishwa Bhat
14 days ago
Although the result on my gdk doesnt exactly match expected behavior:
remote: GitLab: Secret detection scan completed with one or more findings.
remote:
remote: Secrets leaked in commit: 05bb3715d16e777f173d85432cc40bbe66aed1c9
remote: - path: t.txt
remote: - line:2 | GitLab Personal Access Token
remote: - line:3 | GitLab Runner Authentication Token
remote:
remote: Secret leaked in blob: e0715f13106b113bc4a28565fb983cd07fea72bd
remote: -- line:2 | GitLab Personal Access Token
(edited)
Vishwa Bhat
14 days ago
It says Secret leaked in blob instead of leaked in commit
Serena Fang
14 days ago
With the dummy content around the secrets, I'm seeing the same behavior as you :+1: Including the "Secret leaked in blob" part :thinking_face: The blob id corresponds to the first committed blob, so the behavior is technically correct, but it may be more helpful to the user if we display the commit id here instead of blob id. WDYT?
Ahmed
14 days ago
With the dummy content around the secrets, I’m seeing the same behavior as you :+1: Including the “Secret leaked in blob” part :thinking_face: The blob id corresponds to the first committed blob, so the behavior is technically correct, but it may be more helpful to the user if we display the commit id here instead of blob id. WDYT?
The behaviour of showing “Secret leaked in blob” may indicate that the blob id was not marked as a blob that we were able to retrieve the git tree entry for.
Can you confirm if the entries object returned include an entry with that specific blob id?
GitLabGitLab
ee/lib/gitlab/checks/secrets_check.rb · e23de34aa44520655d31d7f04c0bab4833b1e641 · GitLab.org / GitLab · GitLab
GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Self-host GitLab on your own servers, in a...
:pray:
1
Serena Fang
14 days ago
Can you confirm if the entries object returned include an entry with that specific blob id?
It looks like the specific blob id isn't in entries :thinking_face:
Ahmed
13 days ago
As far as I understand, GetTreeEntries RPC uses git cat-file command under the hood. If you run the following command:
git cat-file --batch-check --batch-all-objects | grep blob
In the folder of the repository you’re trying to push from, are you able to see the blob id for the blob in question? :eyes:
Serena Fang
13 days ago
The blob id does appear when I run that command:
git push
remote: Secret leaked in blob: 343584dd47a852f0bd18bab5b824ddf061699443
remote: -- line:3 | GitLab Personal Access Token
git cat-file --batch-check --batch-all-objects | grep blob
343584dd47a852f0bd18bab5b824ddf061699443 blob 68
Ahmed
12 days ago
Good. Now if you do the following:
git cat-file -p REPLACE_WITH_COMMIT_SHA
And replace REPLACE_WITH_COMMIT_SHA with the acutal sha of the commit you know the blob belongs to, you should see an output that looks something like the example below:
tree 21f0bb2d6574d12c8fae1758a17c1ff834b632a0
parent efed8c9cc78bf4049975aa75622adf434569500e
author Ahmed Hemdan <ahemdan@gitlab.com> 1713986576 +0200
committer Ahmed Hemdan <ahemdan@gitlab.com> 1713986576 +0200
Copy the sha in front tree and try the same command but with the new value:
git cat-file -p REPLACE_WITH_TREE_SHA
Do you see the same blob object in the output? I’m trying to pin down whether the problem is with GetTreeEntries RPC or if it’s something different. (edited)
Serena Fang
12 days ago
I see the blob in the output after following those steps.
Secret leaked in blob: bbd7a6c121c587d04e30c7b324ec28400869222b
remote: -- line:2 | GitLab Personal Access Token
________
% git log
commit ae548eeadbdfafafe45b14a18a04dcd9b1ae0d38
Author: Serena Fang <sfang@gitlab.com>
Date: Wed Apr 24 11:09:46 2024 -0500
commit 1
__________
% git cat-file -p ae548eeadbdfafafe45b14a18a04dcd9b1ae0d38
tree ca26d97e688d37e13fc6e7e71110e2680cdd824a
----------
% git cat-file -p ca26d97e688d37e13fc6e7e71110e2680cdd824a
100644 blob bbd7a6c121c587d04e30c7b324ec28400869222b test.txt
Ahmed
12 days ago
It’s weird that it’s showing in the output of git cat-file but not in the entries object. I wonder if you were looking at the set of entries returned for the correct commit? If you’re able to share the repository somewhere, it would be helpful for me to have a look at sometime early next week.
Serena Fang
9 days ago
Testing was done on my local gdk so I'm not sure how to share -- here are steps to reproduce:
Create a file, add a secret with some filler text around it, commit it
Edit the file, add another secret, commit it
Expected behavior:
Secret leaked in commit abc123
test.txt:1 | glpat
Secret leaked in commit def789
test.txt:2 | glrt
Actual behavior:
Secret leaked in commit def789
test.txt:1 | glpat
test.txt:2 | glrt
Secret leaked in blob 1a2b3c
test.txt:1 | glpat
Ahmed
8 days ago
I will try to replicate this sometime later today if I get the time, but I have a hunch this may be due to how GetTreeEntries work. Let me check and circle back to you.
Steps to reproduce
Example Project
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Edited by Serena Fang