Python error during backoff handing in some cases

Problem

Prior to starting a test, the worker-entry script waits for the scan target to become available using an exponential back off module. In some cases, exceptions that are raised are not compatible with the back off module (don't extend from BaseException) causing a python error that is not actionable by the end user.

A customer would run into this issue if the hostname doesn't resolve.

Excerpt from worker-entry debug log:

2022-06-29 21:16:06 [INF] API Security: Backing off 52.1 seconds afters 10 tries
2022-06-29 21:16:58 [WAR] API Security: Waiting for url 'http://vulnapp:8080', failed: catching classes that do not inherit from BaseException is not allowed
2022-06-29 21:16:58 [DEB] API Security: Waiting for url 'http://vulnapp:8080', failed: catching classes that do not inherit from BaseException is not allowed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 162, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.6/http/client.py", line 1273, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 982, in send
    self.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7eff07f7a518>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='vulnapp', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7eff07f7a518>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/backoff/_sync.py", line 43, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/worker_entry/main.py", line 925, in _wait_url
    return requests.get(url, verify=False)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='vulnapp', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7eff07f7a518>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/worker_entry/main.py", line 929, in wait_for_url
    ret = _wait_url(url)
  File "/usr/local/lib/python3.6/site-packages/backoff/_sync.py", line 95, in retry
    except exception as e:
TypeError: catching classes that do not inherit from BaseException is not allowed
2022-06-29 21:16:58 [ERR] API Security: Error waiting for target 'http://vulnapp:8080' to become available.

Proposal

The issue stems from providing the name of a regular class instead of an exception to the on_exception annotation.

  1. Combine multiple on_exception annotations into one
  2. Verify all named exceptions are actually exceptions
  3. Add additional messages to _give_up
  4. Add additional tests for this feature (bug was not caught by unit tests)
  5. Publish new version of container
Edited by Michael Eddington