Discussion: Categories of Fuzzing
Concern
We are currently using the term "Coverage-Guided Fuzzing" as a general category to differentiate between API fuzzing and fuzzing stand-alone/direct targets.
From a technical perspective, coverage-guided fuzzing is a general technique that can be used as feedback in both API fuzzing and stand-alone/direct fuzzing targets. From a marketing or user-facing perspective, we are differentiating two distinct GitLab features using the terms Coverage-guided fuzzing
and API Fuzzing
, which have overlapping technical definitions. Confusion may be caused when using marketing terms and technical descriptions in the same context.
The Venn Diagram below demonstrates the overlap in terminology:
Label | Type | Description |
---|---|---|
T: Coverage |
Technical | Fuzzers that use coverage as feedback during fuzzing |
M: Coverage |
~Marketing | *Generally, the group of fuzzers libfuzzer /go-fuzz /honggfuzz /afl /* |
T: API Fuzzing |
Technical | When APIs are the targets being fuzzed |
M: API Fuzzing |
Marketing | When APIs are the targets being fuzzed |
* Note that using coverage metrics for feedback is not technically required for each of the tools that are generally described as 'coverage-guided'. Other feedback metrics may be used, or none.
Side Note
I do not think that getting rid of the term "Coverage-guided fuzzing" for a more technically-clear term would be the correct course of action. It is an established term that will clearly categorize our offerings.
Sub Concern
This is a minor concern - if we use Coverage-Guided Fuzzing
, which technically speaks to one type of feedback metric, how would we market new feedback mechanisms/techniques that fit into the same, broad, marketing category?
Proposal
A solution to the overlap in terminology may be to clearly define how to technically describe both marketing terms in different contexts.
Example 1:
GitLab supports API Fuzzing and Coverage-Guided Fuzzing
Example 2:
API Fuzzing may use coverage metrics as feedback during fuzzing to increase efficiency
Example 3 - This statement would work with API fuzzing and Coverage-guided fuzzing
Feedback-guided fuzzing uses metrics obtained from the fuzz target via instrumentation to efficiently explore paths within the target program.
Example 4
GitLab uses coverage metrics in Coverage-Guided fuzzing and API fuzzing
Example 5
GitLab uses coverage metrics in stand-alone/direct fuzzing (Coverage-Guided fuzzing) and API fuzzing