DAST Scan always runs out of memory even when setting -Xmx flag
Summary
DAST Scan constantly runs out of memory no matter what I set the heap size to using the -Xmx flag. It will always fail at the same point after 324 minutes no matter how much memory I allocate to the Java heap space.
Steps to reproduce
Run a full scan against my company's website using the DAST template
What is the current bug behavior?
The scan fails due to running out of memory and no report is uploaded
What is the expected correct behavior?
The scan should complete and the pass the stage in the pipeline, as well as uploading the report
Relevant logs and/or screenshots
[zap.out] Found Java version 1.8.0_242
[zap.out] Available memory: 15909 MB
[zap.out] Using JVM args: -Xmx10000m
...
...
[zap.out] 18789342 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - Checkpoint start
[zap.out] 18789342 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - checkpointClose start
[zap.out] 18789378 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - checkpointClose synched
[zap.out] 18789390 [HSQLDB Timer @10d59282020-06-22 19:49:20,183 I/O error: HTTPConnectionPool(host='localhost', port=49666): Max retries exceeded with url: http://zap/OTHER/core/other/messagesHar/?apikey= (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
ERROR HTTPConnectionPool(host='localhost', port=49666): Max retries exceeded with url: http://zap/OTHER/core/other/messagesHar/?apikey= (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
Total of 20 URLs
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=49666): Max retries exceeded with url: http://zap/OTHER/core/other/messagesHar/?apikey= (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/zap/zap_full_scan_original.py", line 458, in main
trigger_hook('handover_to_dast', zap)
File "/zap/zap_common.py", line 111, in trigger_hook
response = hook_fn(*args_list, **kwargs)
File "/zap/custom_hooks.py", line 55, in handover_to_dast
scanned_resources = self.zaproxy.scanned_resources()
File "/zap/zaproxy.py", line 86, in scanned_resources
message_json = self.messages_har()
File "/zap/zaproxy.py", line 116, in messages_har
message = self.zap.core.messages_har()
File "/usr/local/lib/python3.6/dist-packages/zapv2/core.py", line 592, in messages_har
return (self.zap._request_other(self.zap.base_other + 'core/other/messagesHar/', params))
File "/usr/local/lib/python3.6/dist-packages/zapv2/__init__.py", line 189, in _request_other
data = self._request_api(url, get)
File "/usr/local/lib/python3.6/dist-packages/zapv2/__init__.py", line 158, in _request_api
response = self.session.get(url, params=query, proxies=self.__proxies, verify=False)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='localhost', port=49666): Max retries exceeded with url: http://zap/OTHER/core/other/messagesHar/?apikey= (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
cp: cannot stat '/zap/wrk/*': No such file or directory
6] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - checkpointClose script done
[zap.out] 18789390 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - dataFileCache commit start
[zap.out] 18789476 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - dataFileCache commit end
[zap.out] 18789507 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - checkpointClose end
[zap.out] 18789507 [HSQLDB Timer @10d59286] INFO hsqldb.db.HSQLDB379AF3DEBD.ENGINE - Checkpoint end - txts: 45132
[zap.out] 19395187 [Thread-12] INFO org.parosproxy.paros.core.scanner.HostProcess - completed host/plugin https://<REDACTED> | SlackerCookieDetector in 629.221s with 344 message(s) sent and 18 alert(s) raised.
[zap.out] 19438227 [ZAP-ProxyThread-7728] ERROR org.zaproxy.zap.ZAP$UncaughtExceptionLogger - Exception in thread "ZAP-ProxyThread-7728"
[zap.out] java.lang.OutOfMemoryError: Java heap space
[zap.out] at java.util.Arrays.copyOf(Arrays.java:3236)
[zap.out] at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
[zap.out] at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
[zap.out] at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
[zap.out] at org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1748)
[zap.out] at org.codehaus.jackson.impl.Utf8Generator._writeStringSegments(Utf8Generator.java:1207)
[zap.out] at org.codehaus.jackson.impl.Utf8Generator._writeLongString(Utf8Generator.java:575)
[zap.out] at org.codehaus.jackson.impl.Utf8Generator.writeString(Utf8Generator.java:550)
[zap.out] at org.codehaus.jackson.impl.Utf8Generator.writeStringField(Utf8Generator.java:262)
[zap.out] at edu.umass.cs.benchlab.har.HarContent.writeHar(HarContent.java:254)
[zap.out] at edu.umass.cs.benchlab.har.HarResponse.writeHar(HarResponse.java:352)
[zap.out] at edu.umass.cs.benchlab.har.HarEntry.writeHar(HarEntry.java:348)
[zap.out] at edu.umass.cs.benchlab.har.HarEntries.writeHar(HarEntries.java:161)
[zap.out] at edu.umass.cs.benchlab.har.HarLog.writeHar(HarLog.java:187)
[zap.out] at edu.umass.cs.benchlab.har.tools.HarFileWriter.writeHarFile(HarFileWriter.java:90)
[zap.out] at org.zaproxy.zap.utils.HarUtils.harLogToByteArray(HarUtils.java:112)
[zap.out] at org.zaproxy.zap.extension.api.CoreAPI.handleApiOther(CoreAPI.java:1497)
[zap.out] at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:544)
[zap.out] at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:499)
[zap.out] at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:335)
[zap.out] at java.lang.Thread.run(Thread.java:748)
cat: /tmp/.X1-lock: No such file or directory
/zap/zap-x.sh: 10: kill: Usage: kill [-s sigspec | -signum | -sigspec] [pid | job]... or
kill -l [exitstatus]