Replace ZAP Python Scripts in DAST

Problem to solve

Currently, control flow is delegated to ZAP Python scripts:

graph LR
   /analyze --> DAST_Python_Entry --> ZAP_Python_Baseline & ZAP_Python_Fullscan & ZAP_Python_Apiscan --> DAST_Python_Exit

Limitations

This has the following limitations:

  • Changing the control flow in ZAP Python is hard. This limits the features we can build, and we have resorted to some nasty ways to get around this (monkey patching, find/replace their code).
  • The ZAP Python scripts aren't well tested, e.g. no unit tests.
  • Using the ZAP Python scripts means we have to use the ZAP weekly image. This is an Ubuntu image, and results in the DAST image being ~1.5GB in size.
  • Requiring the ZAP Docker image for our source limits our ability to test, run locally, substitute the ZAP server, etc.
  • Our IDEs can't resolve references, because they don't exist in our code repository.
  • It's hard to understand how DAST works. You need to find the ZAP source code to piece together everything. This is hard for community contributors, and for new developers.
  • Building a feature for one scan type should work for all scan types.

Proposal

We move the functionality from ZAP Python scripts into DAST.

graph LR
   /analyze --> DAST_Python

Out of scope

  • Changing the way the DAST product works. This is technical debt, behaviour should not be changed.
  • There is an argument that the /analyze script is of no value. This will not be changed as part of this work.
  • We should be able to use the zapv2 client library. If the ZAP Python scripts can use it now, so can the new Python scripts.

Limitations

  • The resulting code will still by Python, which isn't an officially supported GitLab language.
  • Any updates that occur in the original ZAP Python scripts, we will be missing out on.

Migration strategy

  • Strangulation approach:
    • We already control the top and bottom of the control flow: ZAP Python scripts are already surrounded
    • Over time, we squeeze the ZAP scripts, moving functionality from the top/bottom into DAST Python
    • Eventually, all code is in DAST Python and ZAP scripts disappear
  • Code that is moved must be already tested
  • No behaviour is changed, so this will not require a major DAST release
  • Need to ensure we don't miss DAST deadlines because of this work

Progress

2 of 32 tasks complete
Migrated Baseline Script Full Scan Script API Scan Script
✅ Parse configuration Parse configuration Parse configuration
Verify /zap/wrk has been mounted Verify /zap/wrk has been mounted Verify /zap/wrk has been mounted
Find random port Find random port Find random port
Load config from file, if exists Load config from file, if exists Load config from file, if exists
Load config from URL, if exists Load config from URL, if exists Load config from URL, if exists
Load progress file, if exists Load progress file, if exists Load progress file, if exists
Configure ZAP Configure ZAP Configure ZAP
Start ZAP Start ZAP Start ZAP
Wait for ZAP to start Wait for ZAP to start Wait for ZAP to start
trigger: zap_started trigger: zap_started trigger: zap_started
Import context, if exists Import context, if exists Import context, if exists
Test can access URL Test can access URL
Load API custom API scripts
Import API definition (start scan)
Update target to be base URL Update target to be base URL Update target to be base URL
Start passive scan Start passive scan
Wait for scan to finish Wait for scan to finish
Start ajax scan Start ajax scan
Wait for scan to finish Wait for scan to finish
Start active scan Start active scan
Wait for scan to finish Wait for scan to finish
Wait for passive scan Wait for passive scan Wait for passive scan
Print number of URLs Print number of URLs Print number of URLs
Print vulnerability rules Print vulnerability rules Print vulnerability rules
Generate config file, if requested Generate config file, if requested Generate config file, if requested
Print alerts Print alerts Print alerts
Write reports Write reports Write reports
Print summary Print summary Print summary
trigger: zap_pre_shutdown trigger: zap_pre_shutdown trigger: zap_pre_shutdown
Stop ZAP Stop ZAP Stop ZAP
trigger: pre_exit trigger: pre_exit trigger: pre_exit
✅ Exit Exit Exit

Common ZAP Script configuration

-h                print this help message
-c config_file    config file to use to INFO, IGNORE or FAIL warnings
-u config_url     URL of config file to use to INFO, IGNORE or FAIL warnings
-g gen_file       generate default config file (all rules set to WARN)
-r report_html    file to write the full ZAP HTML report
-w report_md      file to write the full ZAP Wiki (Markdown) report
-x report_xml     file to write the full ZAP XML report
-J report_json    file to write the full ZAP JSON document
-a                include the alpha scan rules as well
-d                show debug messages
-P                specify listen port
-D                delay in seconds to wait for passive scanning 
-i                default rules not in the config file to INFO
-l level          minimum level to show: PASS, IGNORE, INFO, WARN or FAIL, use with -s to hide example URLs
-n context_file   context file which will be loaded prior to scanning the target
-p progress_file  progress file which specifies issues that are being addressed
-s                short output format - dont show PASSes or example URLs
-T                max time in minutes to wait for ZAP to start and the passive scan to run
-z zap_options    ZAP command line options e.g. -z "-config aaa=bbb -config ccc=ddd"
--hook            path to python file that define your custom hooks

API specific configuration

-t target         target API definition, OpenAPI or SOAP, local file or URL, e.g. https://www.example.com/openapi.json
-f format         either openapi or soap
-S                safe mode this will skip the active scan and perform a baseline scan
-O                the hostname to override in the (remote) OpenAPI spec

Baseline specific configuration

-t target         target URL including the protocol, e.g. https://www.example.com
-m mins           the number of minutes to spider for (defaults to no limit)
-I                do not return failure on warning
-j                use the Ajax spider in addition to the traditional one

Full scan specific configuration

-t target         target URL including the protocol, e.g. https://www.example.com
-m mins           the number of minutes to spider for (defaults to no limit)
-j                use the Ajax spider in addition to the traditional one

References

Internal Google Doc

Edited by Cameron Swords