Remove GetTreeEntries from SPP

What does this MR do and why?

This drops the use of GetTreeEntries() RPC in Secret Push Protection which is used to correlate a secret finding to a commit SHA and a file path in favour of using the newly introduced attribute commit_id that is included now with all FindChangedPaths() RPC responses for each ChangedPath entry when the request is a CommitRequest (which is the case for SPP).

To achieve this, we build out a map with all payloads and their changed paths (including the associated commit SHA for each individual ChangedPath entry), and then while we build out the message that we display to the user, we pull the commit SHA and the file path directly from that map instead of calling GetTreeEntries() for each commit as was the case earlier.

This performance optimization would potentially reduce the time it takes to complete a Secret Push Protection scan as we:

• Call ListAllCommits() RPC (once per scan)
• Call FindChangedPaths() RPC (once per scan)
• Call DiffBlobs() RPC (once per 50 changed paths)

The old workflow (before this change) was:

• Call ListAllCommits() RPC (once per scan)
• Call FindChangedPaths() RPC (once per scan)
• Call DiffBlobs() RPC (once per 50 changed paths)
• Call GetTreeEntries() RPC (once per commit, e.g. for +100 commits, we make +100 calls to GetTreeEntries)

Diagrams –– Before

Sequence Diagram

sequenceDiagram
    autonumber
    actor User
    User->>+Workhorse/GitLab Shell: git push
    Workhorse/GitLab Shell->>+Gitaly: tcp/ssh
    Gitaly->>+Gitaly: PostReceivePack/SSHReceivePack
    Gitaly->>+Gitaly: git-receive-pack
    Gitaly->>+Gitaly: PreReceiveHook
    Gitaly->>+Rails: grpc
    Note over Gitaly, Rails: invokes /internal/allowed endpoint
    Rails->>+Rails: GitLab::GitAccess
    Rails->>+Rails: EE::GitLab::Checks::ChangesAccess
    Note over Rails: runs Gitlab::Checks::SecretsCheck
    break when special commit message flag is found
        Rails->>+Gitaly: push check skipped
        Gitaly->>+Workhorse/GitLab Shell: outcome of push
        Workhorse/GitLab Shell->>+User: outcome of push
    end
    break when push option is passed
        Rails->>+Gitaly: push check skipped
        Gitaly->>+Workhorse/GitLab Shell: outcome of push
        Workhorse/GitLab Shell->>+User: outcome of push
    end
    Rails->>+Gitaly: ListCommits or ListAllCommits
    Note over Gitaly, Rails: depends on quarantine directory existence
    Gitaly->>+Rails: grpc
    Rails->>+Gitaly: FindChangedPaths
    Note over Gitaly, Rails: returns all changed files for new commits
    Gitaly->>+Rails: grpc
    Rails->>+Gitaly: DiffBlobs
    Note over Gitaly, Rails: returns all diff patches for changed files
    Gitaly->>+Rails: grpc
    Rails->>+gitlab-secret_detection: gitlab-secret_detection::Scan
    alt no secret detected
        gitlab-secret_detection->>+gitlab-secret_detection: scan blob
        gitlab-secret_detection->>+Rails: success
        Rails->>+Gitaly: accept - no secret detected
    else scan timeout
        gitlab-secret_detection->>+gitlab-secret_detection: scan blob
        gitlab-secret_detection->>+Rails: fail - timeout
        Rails->>+Gitaly: accept - scan timeout
    else secret detected
        gitlab-secret_detection->>+gitlab-secret_detection: scan blob
        gitlab-secret_detection->>+Rails: fail - secret found
        Rails->>+Gitaly: GetTreeEntries
        Note over Gitaly, Rails: retrieves blobs' file path and commit sha
        Gitaly->>+Rails: grpc
        Rails->>+Rails: Format Response
        Rails->>+Gitaly: reject - secret detected
    end
    Gitaly->>+Workhorse/GitLab Shell: outcome of push
    Workhorse/GitLab Shell->>+User: outcome of push

Please check number 30 and number 31 for the old workflow.

Workflow Diagram

Diagrams –– After

Sequence Diagram

sequenceDiagram
    autonumber
    actor User
    User->>+Workhorse/GitLab Shell: git push
    Workhorse/GitLab Shell->>+Gitaly: tcp/ssh
    Gitaly->>+Gitaly: PostReceivePack/SSHReceivePack
    Gitaly->>+Gitaly: git-receive-pack
    Gitaly->>+Gitaly: PreReceiveHook
    Gitaly->>+Rails: grpc
    Note over Gitaly, Rails: invokes /internal/allowed endpoint
    Rails->>+Rails: GitLab::GitAccess
    Rails->>+Rails: EE::GitLab::Checks::ChangesAccess
    Note over Rails: runs Gitlab::Checks::SecretsCheck
    break when special commit message flag is found
        Rails->>+Gitaly: push check skipped
        Gitaly->>+Workhorse/GitLab Shell: outcome of push
        Workhorse/GitLab Shell->>+User: outcome of push
    end
    break when push option is passed
        Rails->>+Gitaly: push check skipped
        Gitaly->>+Workhorse/GitLab Shell: outcome of push
        Workhorse/GitLab Shell->>+User: outcome of push
    end
    Rails->>+Gitaly: ListCommits or ListAllCommits
    Note over Gitaly, Rails: depends on quarantine directory existence
    Gitaly->>+Rails: grpc
    Rails->>+Gitaly: FindChangedPaths
    Note over Gitaly, Rails: returns all changed files for new commits
    Gitaly->>+Rails: grpc
    Rails->>+Rails: Populate PayloadPathsLookupMap with commit sha/file path
    Rails->>+Gitaly: DiffBlobs
    Note over Gitaly, Rails: returns all diff patches for changed files
    Gitaly->>+Rails: grpc
    Rails->>+gitlab-secret_detection: gitlab-secret_detection::Scan
    alt no secret detected
        gitlab-secret_detection->>+gitlab-secret_detection: scan blob
        gitlab-secret_detection->>+Rails: success
        Rails->>+Gitaly: accept - no secret detected
    else scan timeout
        gitlab-secret_detection->>+gitlab-secret_detection: scan blob
        gitlab-secret_detection->>+Rails: fail - timeout
        Rails->>+Gitaly: accept - scan timeout
    else secret detected
        gitlab-secret_detection->>+gitlab-secret_detection: scan blob
        gitlab-secret_detection->>+Rails: fail - secret found
        Rails->>+Rails: Use PayloadPathsLookupMap to retrieve commit sha/file path
        Rails->>+Rails: Format Response
        Rails->>+Gitaly: reject - secret detected
    end
    Gitaly->>+Workhorse/GitLab Shell: outcome of push
    Workhorse/GitLab Shell->>+User: outcome of push

Please check number 19 and number 32 for the new workflow.

Workflow Diagram

Feature Flags

This change is implemented behind the drop_get_tree_entries_from_spp feature flag. It's a gitlab_com_derisk flag.

References

Resolves #477389 (closed).

Screenshots

Before	After

Yes, it's basically the same output. Nothing changed in the user experience.

How to set up and validate locally

1. With master checked out, in your GDK: add an Ultimate license, then enable Secret Push Protection in a project.
2. In the project, add a commit including a secret, e.g. glpat-12345678901234567890 (dummy example) and try to push.
3. Observe that the secret is detected and the push is blocked with a message including file path, line number and commit SHA.

Secret push protection found the following secrets in commit: [COMMIT_SHA]
-- [FILENAME]:[LINE_NUMBER] | [TOKEN_TYPE] 

4. Switch to remove-get_tree_entries-from-spp branch, and restart your GDK to make sure code is also reloaded.
5. Try to push the same file with the secret again.
6. Your push should still be blocked, and display the same information as in the example above.

MR acceptance checklist

I have evaluated this MR against the MR acceptance checklist.