We talked recently about having a way to deploy Defend applications (WAF, NetworkPolicies, etc.) without a pipeline (#211505 (closed)), but we’ll probably stick to a pipeline version anyway, especially with the upcoming Managed Apps V2.
This is why I don't think we should move away from the pipeline:
The pipeline remains the SSOT. If anything goes wrong, users know where to get started for debugging
The pipeline is a performant, scalable, and secure way to execute external code.
If we start having agents or services, we need to bundle them for shipping, monitor them, and so on. If these services are stateful, we need a way to backup them, we need to document how to restore the data. They can't ship on the same servers as GitLab. If they go crazy, they will tear down the GitLab instance with them. They also augment the attack surface of our product.
If we don't use the pipeline, we'll pretty much reinvent the wheel to create a system capable of running tasks independently. The input is always the repo (at the time of the current coming being assessed) and the output a JSON report.
Some jobs might take more than the default runner timeout (1h), so it would seem obvious to create something outside of the pipeline. I don't think we should do that either. We should instead think about a way to run background jobs, that don't block the pipeline. In the end, these jobs will have to run a frozen version of the repo (a commit), so we'll need to connect the dots between the scans and the repo workflow anyway. Maybe that means we'll run only heavy tasks like fuzzing for tags, or once a week on master.
I don't think we have enough Engineering resources in Secure to manage this kind of product anyway, the effort is far from being negligible.
A clarification here, what's the goal for getting away from a pipeline?
A: Is it about not having a specific code change (wanting to be able to run on a schedule or arbitrarily outside of a code change)
This seems like something scheduled pipelines and ghost pipelines could be the answer for.
B: Is this about performance and not blocking git/commit/merge workflows?
This is a whole different set of problems that incremental scanning could solve.
C: Is it about being able to run scans and prevent things from hitting the server (secret detection is a great example of this) or that are outside of standard code changes (like security or docker images, etc.
@tmccaslin Option A. Scheduled pipelines would be what I thought of. @cblake suggested that there might be a reason why we don't promote this. If technically feasible, then customers could periodically scan their code and uncover issues due to newly discovered vulnerabilities.
This something we're going to make more clear for Secret Detection and SAST. I'm more than happy to leverage this issue to push out a doc update on using Scheduled pipelines to do regular scans.