Support Continuous Fuzzing for Coverage Guided Fuzzing
Problem to solve
A robust fuzzing job can be a long running task. Additionally, customers may wish to fuzz a target indefinitely until a crash is found. For mature projects, this could be hours or days. Longer duration fuzz testing will find additional bugs and vulnerabilities.
In the above scenario, running a fuzzing job in a pipeline does not make sense, since the pipeline will never complete or will timeout after a certain period.
User experience goal
Consider how, where, and when to surface results from continuous fuzz testing as compare to results from a pipeline. Since there is no "done" to a continuous fuzz test job, we don't have the normal "end of a pipeline" step to collect and process results.
- Idea: Consider introducing a new screen where continuous fuzz testing results are collected and then can be manually promoted to vulnerabilities/findings after a user has reviewed them. This would be similar to some of the earlier designs we saw where all fuzz testing results were on their own screen.
A goal is that the same approach & UX can be used for both coverage-guided & API fuzzing.
Proposal
GitLab should support fuzzing jobs that run outside of a pipeline, and run until one of the following occurs:
- Indefinitely
A crash is foundA set amount of time is past
Note: Provide a variable to let the users choose the other mode if so desired.
When setting up continuous fuzz testing jobs, require a user to indicate a specific job in their pipeline to use as the fuzz testing target. This would allow GitLab to not have to guess or build complicated auto-detection logic to find out what to fuzz for and users would likely already have this configured since they were using normal fuzz testing already. - We can revisit this in the future too, but want to simplify this seemingly complex problem.
Allow the user to start the continuous job:
- Manually triggered and specify needed configuration options.
Continuous fuzz testing should not block a pipeline job that is started as part of a commit.
- This is so that the CI pipeline will actually finish, since continuous fuzz testing wont ever finish
Further details
For the first iteration of continuous fuzz testing, we want to focus on:
- Migrating existing functionality GitLab acquired, rather than build from scratch where possible
- Iterating small with a "boring solution" that we can expand on and improve more later.
Questions to answer:
- What scenarios should cause a continuous fuzzing job to stop
- How can a user stop a long running fuzz job?
- When should new builds be fuzzed, if a previous build is being fuzzed indefinitely
- How does GitLab save and present results for a job that is still running?
- What can we re-use from on-demand scans?
Permissions and Security
Documentation
Documentation needs to be updated to describe:
- What continuous fuzz testing is & the problems it addresses
- Discussion of the differences between continuous fuzzing & normal fuzzing
- Configuring continuous fuzz testing
- An example project
- Screenshots / visuals where applicable
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Is this a cross-stage feature?
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.