Skip to content

Support `git` and `url` passthroughs

Ahmed Hemdan requested to merge process-git-and-url-passthroughs-additionally into master

What does this MR do?

This merge request adds support for git and url passthroughs via ruleset module's ProcessPassthroughs method.

Background

We initially supported file and raw passthroughs, but using ruleset module will allow us to support git and url as well.

Our support for file and raw was very basic, but with the move to use ruleset module for handling passthroughs, we will have:

  • Support for passthrough chains with two modes: append (doesn't work with git passthrough), overwrite.
  • Support for variable interpolation in passthroughs to include environment variables.
  • Support for setting a timeout when loading a passthrough.
  • Support for passthrough config validation (toml).

Testing

As part of this merge request, I have created a number of projects to use in tests:

Synthesizing a default configuration

Passthrough Fixture Description
file synthesize-file-passthrough Replaces the default ruleset configuration with a gitleaks.toml file stored in the same project. The new file only detects glpat tokens, so the glptt token in file-with-no-longer-used-rule.txt will no longer be detected.
raw synthesize-raw-passthrough Replaces the default ruleset configuration with inline configuration that only detects glpat tokens, so the glptt token in file-with-no-longer-used-rule.txt will no longer be detected.
git synthesize-git-passthrough Replaces the default ruleset configuration with a remote git configuration that only detects Stripe API tokens, so the glptt token in file-with-no-longer-used-rule.txt will no longer be detected.
url synthesize-url-passthrough Replaces the default ruleset configuration with a url configuration that only detects Stripe API tokens, so the glptt token in file-with-no-longer-used-rule.txt will no longer be detected.

Extending default configuration

Passthrough Fixture Description
file extend-with-file-passthrough Extends the default ruleset configuration with a extended-gitleaks-config.toml file stored in the same project. The new file adds a specific glpat token to the [allowlist], in such a way that a regular glpat token would be detected but the one in file-with-allowlisted-secret.txt is not detected.
git extend-with-git-passthrough Extends the default ruleset configuration with a remote git configuration. The new file adds a specific glpat token to the [allowlist], in such a way that a regular glpat token would be detected but the one in file-with-allowlisted-secret.txt is not detected.
url extend-with-url-passthrough Extends the default ruleset configuration with a remote git configuration. The new file adds a specific glpat token to the [allowlist], in such a way that a regular glpat token would be detected but the one in file-with-allowlisted-secret.txt is not detected.

I think it's important to create a number of demos based on these projects under security-products/demos.

In addition, I have confirmed following use cases work (as outlined in the table above) to make sure the issue's proposal is satisfied:

  • Extend default configuration with a git passthrough.
  • Extend default configuration with a url passthrough.

Note: I created security-products/tests/secrets-passthrough-git-and-url-test to store the configuration used in the following fixtures:

  • synthesize-git-passthrough
  • synthesize-url-passthrough
  • extend-with-git-passthrough
  • extend-with-url-passthrough

And I have also made sure to leave a comment in the README to indicate it shouldn't be removed or updated without a corresponding merge request in secrets analyzer.

What are the relevant issue numbers?

Does this MR meet the acceptance criteria?

Edited by Lucas Charles

Merge request reports