Investigate why -o secret_push_protection.skip_all wasn't respected for merge-train project

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem Description

The merge-train project is used by several pipelines across different projects in gitlab-org group to sync and mirror between the canonical repository and the security and FOSS mirrors. This project encapsulates a shell script into a docker image that gets executed in CI jobs.

As part of the sync/mirror process, a few git push commands are executed. When those push commands are executed, secret push protection is invoked. In https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20491, the groupsecret detection and grouprelease-and-deploy had discussed the need to skip secret push protection scans in similar automated workflows (please see the discussion for more details on the why).

One of the skip options available is to pass a git push option, i.e. -o secret_push_protection.skip_all to the git push command. The grouprelease-and-deploy team had updated the shell script to use the push option, but Secret Push Protection was still being executed in CI jobs (see example).

This issue aims to investigate why the push option was not respected in the merge-train project. Please note that, most of the CI jobs were running on the merge-train project in the https://ops.gitlab.net instance and not the https://gitlab.com instance. We have confirmed the project in both instances had the latest code changes.

Timeline

Below is a brief timeline of the incident, and how it was resolved.

• A report emerged that commits from canonical master to security mirror are failing with 500 errors in CI pipelines.
• A brief investigation had identified this to be due to SPP scans timing out as calls to GetTreeEntries() are taking too long.
• The team re-iterated that such an automated workflow is an invalid use case for SPP, and suggested skipping the scan.
• The grouprelease-and-deploy had updated the shell script to skip SPP scans in merge-train!64 (merged).
• Despite confirming the changes were successfully built into the docker image, SPP scans were still invoked in the CI jobs.
• Another merge request was submitted to increase debugging level. It resulted in a different issue with OpenSSL.
• The team had identified that push_options isn't passed to /api/:version/internal/allowed (see Screenshot 1 below).
• While that investigation continued, the grouprelease-and-deploy decided to use the other skip option to unblock mirroring.
• The team had identified that push_options was also missing from initial git-receive-pack request (see Screenshot 2 below).
• The team reached out to groupsource code to help with investiating push_options behaviour.
• The team confirmed push_options are passed properly for other projects (see Screenshot 3 below).
• As the issue with OpenSSL continued, the grouprelease-and-deploy decided to disable SPP to unblock mirroring.
• The issue with OpenSSL had been identified and fixed in merge-train!67 (merged).

NOTE: The skip option via the commit message was never tested. The team asked grouprelease-and-deploy to test this whenever they could, so we can identify the root cause and understand if this only affects push options or the entire Secret Push Protection functionality.

Screenshots

Below are a few screenshots that were used in debugging during the incident.

Screenshot 1

Screenshot 2

Screenshot 3

References

Failing CI jobs (on https://ops.gitlab.net) can be found below:

• https://ops.gitlab.net/gitlab-org/merge-train/-/jobs/20387136
• https://ops.gitlab.net/gitlab-org/merge-train/-/jobs/20387447

And the associated correlation IDs (can be checked via the support dashboard):

• 01K5RHY6NBP388F0AWB6CGCTAX
• 01K5RHY2MS1TC53AN4PR4HX0SZ