Gain confidence when changing DAST by adding more tests
Problem to solve
A recent incident gitlab-com/gl-infra/production#1246 (closed) highlighted some issues related to our testing of DAST.
Specifically,
- tests that were breaking were not failing on CI gitlab-org/security-products/dast!56 (merged)
- tests have been merged to use
bash_unit
to make it easier to read/write tests gitlab-org/security-products/dast!57 (merged) - there are significant gaps in features we support and tests we have #29601 (closed)
Proposal
This issue does not attempt to fix all of the gaps identified so far in our testing process.
Tests to add:
- Legacy endpoints (baseline, fullscan)
- Configuration (-r, -w, -x)
Note that options -a, -z, -m, -n will not been tested as part of this issue.
In order to get these tests passing, the output of DAST will be changed to be deterministic. This means that keys in the JSON output will be in alphabetical order, and arrays will be sorted. This also aligns with the Robustness principle.
Edited by Cameron Swords