DAST scan is looping over GET request http://zap/OTHER/core/other/messageHar/?id=X
Summary
First of all apologies for being vague, the behaviour that is described in the issues is not consistent and only happens like 1 in 10 runs and I'm not quite sure what triggers this.
The observation is, DAST scan jobs at times sends thousands of GET http://zap/OTHER/core/other/messageHar/?id=X
request causing the DAST job to keep running even after the scan is competed in some case (pipeline 1). This also results in Job's log exceeded limit of 4194304 bytes
error in job console. Here are few instances.
- https://gitlab.com/gitlab-org/gitlab/-/jobs/647426598
- https://gitlab.com/gitlab-org/gitlab/-/jobs/651186750
Steps to reproduce
Set up a DAST job with target application that is something large as gitlab and execute the job. Again, this behaviour is not consistent and does not happens all the time. A typical job configuration is as below. DAST configuration for the above jobs can be found in the reports.gitlab-ci.yml
file in the commits.
DAST-fullscan-5/5:
variables:
image:
name: "registry.gitlab.com/gitlab-org/security-products/dast:$DAST_VERSION"
variables:
# DAST_USERNAME: "root"
DAST_USERNAME: "root"
DAST_USERNAME_FIELD: "user[login]"
DAST_PASSWORD_FIELD: "user[password]"
DAST_FULL_SCAN_ENABLED: "true"
DAST_SPIDER_MINS: 0
DAST_VERSION: 1
DAST_ZAP_CLI_OPTIONS: "-Xmx6144m -config database.recoverylog=false"
DAST_EXCLUDE_RULES: "41,..,90033"
script:
- 'export DAST_WEBSITE="${DAST_WEBSITE:-$(cat environment_url.txt)}"'
- 'export DAST_AUTH_URL="${DAST_WEBSITE}/users/sign_in"'
- 'export DAST_PASSWORD="${..}"'
- 'export DAST_AUTH_EXCLUDE_URLS="${DAST_WEBSITE}/groups/*/-/edit,${DAST_WEBSITE}/profile/two_factor_auth"'
- '/analyze -t $DAST_WEBSITE -d'
Example Project
- https://gitlab.com/gitlab-org/gitlab/-/jobs/647426598
- https://gitlab.com/gitlab-org/gitlab/-/jobs/651186750
What is the current bug behavior?
DAST scan job keeps running even after the scans are finished.
What is the expected correct behavior?
DAST scan job should finish up soon after runnings the scans.
Relevant logs and/or screenshots
The request looks like below:
2020-07-20 19:05:17,954 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7140&apikey= HTTP/1.1" 200 35737
2020-07-20 19:05:17,957 Starting new HTTP connection (1): localhost:44521
2020-07-20 19:05:17,959 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7140&apikey= HTTP/1.1" 200 35737
2020-07-20 19:05:17,962 Starting new HTTP connection (1): localhost:44521
2020-07-20 19:05:17,963 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7140&apikey= HTTP/1.1" 200 35737
2020-07-20 19:05:17,966 Starting new HTTP connection (1): localhost:44521
2020-07-20 19:05:17,968 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7140&apikey= HTTP/1.1" 200 35737
2020-07-20 19:05:17,971 Starting new HTTP connection (1): localhost:44521
2020-07-20 19:05:17,972 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7141&apikey= HTTP/1.1" 200 35793
2020-07-20 19:05:17,975 Starting new HTTP connection (1): localhost:44521
2020-07-20 19:05:17,977 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7141&apikey= HTTP/1.1" 200 35793
2020-07-20 19:05:17,980 Starting new HTTP connection (1): localhost:44521
2020-07-20 19:05:17,981 http://localhost:44521 "GET http://zap/OTHER/core/other/messageHar/?id=7141&apikey= HTTP/
Job's log exceeded limit of 4194304 bytes.
Implementation plan
-
Use the ZAP database to retrieve messages by ID -
Use the ZAP database to retrieve scanned messages -
Create Message and Messages first class types -
Only run the memory end to end test on master -
Verify that there is no memory error in ZAP logs run by the memory test java.lang.OutOfMemoryError: Java heap space
-
Write an integration test for code that accesses the ZAP database