more retries on tls scans, faster fetching of results, more debug info

parent 2c4d1b2d
Pipeline #19261578 passed with stages
in 40 minutes and 11 seconds
# Scanning policy
Failmap tries to scan with all scanners every day. This means the map shows new results on a daily basis.
## What does failmap scan?
Failmap scans the following:
Daily scans:
- Endpoint discovery (looking for new endpoints and cleaning up old ones)
- HTTP headers
- Endpoints missing encryption
Weekly scans:
- New subdomains (using various methods)
- TLS quality using Qualys SSL Labs
**Endpoint discovery**
Failmap tries to auto-discover endpoints for urls. A normal website today has about four endpoints:
- One on IPv4, port 80 that redirects to port 443. Example:
- One on IPv4, port 443 that contains the website. Example:
- One on IPv6, port 80 that redirects to port 443. Example:
- One on IPv6, port 443 that contains the website. Example:
Since it's possible to host a website on any port, failmap also scans for the existence of websites on well known
(official) alternative ports such as 8080.
The existence of an endpoint in itself is not rated. This is implicit: the more endpoints, the more risk.
**HTTP Headers**
## Special cases
Since failmap is completely automated, there are some special cases that could help improving the result.
These are:
**No HSTS header requirement when there are only encrypted endpoints available.**
Only if there are no unencrypted endpoints available on the url, the HSTS header is not required.
Many products do not use the HSTS header as they don't provide an unsecured endpoint. Those products usually also
use their own client which (should) only try to communicate securely. Forcing HSTS everywhere would require every
vendor to add a header that will be removed soon (and that does not provide additional security for it's clients).
Since it's impossible to distinguish websites from specific services running over HTTPS, we're allowing to omission
of the HSTS header when there are only secure endpoints available on the url.
The HSTS header, as HTTPS preloading are quickfixes to help migrate to a more secure internet. Eventually browsers
will not contact websites on port :80 (unencrypted) anymore. For the time being this header and possibly preloading
can help against downgrading in various scenario's.
**X-Frame Options is ignored on redirects.**
Thanks to: antoinet.
From the terminology used in RFC 7034,
The use of "X-Frame-Options" allows a web page from host B to declare that its content (for example, a
button, links, text, etc.) must not be displayed in a frame (<frame> or <iframe>) of another page (e.g.,
from host A). This is done by a policy declared in the HTTP header and enforced by browser implementations
as documented here.
The X-Frame-Options HTTP header field indicates a policy that specifies whether the browser should render
the transmitted resource within a <frame> or an <iframe>. Servers can declare this policy in the header of
their HTTP responses to prevent clickjacking attacks, which ensures that their content is not embedded
into other pages or frames.
Similarly, since a redirect is a flag not to render the content, the content can't be manipulated.
This also means no X-XSS-Protection or X-Content-Type-Options are needed. So just follow all redirects.
## Supported scans
| Scan | Port(s) | IPv Support | Protocols | Rate limit | Rotation |
| :------------------ | :---------- | :---------- | :-------- | :--------- | :--------- |
| DNS | A/AAAA | - | DNS | No | Not yet automated |
| Endpoint discovery | Defaults | 4 | http(s) | No | Per 3 days |
| TLS (qualys) | 443 | 4, 6 | TLS | 1/minute | Per 3 days |
| Headers | Any http(s) | 4 | http(s) | No | Daily |
| Screenshots | Any http(s) | 4 | http(s) | 1 thread | Not yet automated |
| Plain HTTPS | Any http(s) | 4 | http(s) | No | Daily |
| DNSSEC | - | - | DNS | No | Daily |
### DNS
The DNS scanner tries to find hostnames using various strategies:
- Brute force on a subdomain list (existing subdomains only)
- Looking at NSEC1 hashes
- Looking at Certificate transparency
Less popular, not fully automated, but also implemented:
- brute forcing dictionaries
- looking in search engines
### Endpoint Discovery
Tries to find HTTP(s) endpoints on standard HTTP(s) ports. A normal website currently has about four endpoints:
- IPv6 port 80, redirect to port 443
- IPv6 port 443, actual website
- IPv4 port 80, redirect to port 443
- IPv4 port 443, actual website
We store them separately as implementation mistakes might occur on any of these endpoints.
### TLS (qualys)
Runs a scan on ssllabs from Qualys and incorporates the result.
### Headers
Contacts an endpoint and verifies HTTP headers for various security settings. (HSTS etc)
### Screenshots
Uses chrome headless to contact a website and make a screenshot for it. This screenshow it displayed next to the results
in the report.
### Plain HTTPS
Checks if a website that only has a site on port 80 also has a secure equivalent. No port-80-only sites should exist.
Checks if the toplevel domain implements DNSSEC correctly. Uses the dotSE scanner which is included.
## Scheduling
Scanners are scheduled as periodic tasks in Django admin. They are disabled by default and might not all be included in
the source distribution. Creating a scan is actually easy. For example:
- General/Name: discover-endpoints
- General/Enabled: Yes
- General/Task: discover-endpoints
- Schedule/Interval: every 3 days
- Arguments/Arguments: ["failmap.scanners.scanner_http"]
- Execution Options/Queue: storage
## Manual scans
### Command line
The Scan command can help you:
failmap scan 'scanner name'
The message returned will tell you what scanners you can run manually. All scanners have the same set of options.
### Admin interface
It's possible to run manual scans, at the bottom of a selection.
Note that this is beta functionality and please don't do this too much as the "priority" scanning queue is not functioning.
You can try out a scan or two, some take a lot of time.
......@@ -148,7 +148,7 @@ def qualys_scan(self, url):
# ex: EOF occurred in violation of protocol (_ssl.c:749)
log.exception("(Network or Server) Error when contacting Qualys for scan on %s", url.url)
# Initial scan (with rate limiting) has not been received yet, so add to the qualys queue again.
raise self.retry(countdown=60, priorty=PRIO_NORMAL, max_retries=30, queue='scanners.qualys')
raise self.retry(countdown=60, priorty=PRIO_NORMAL, max_retries=100, queue='scanners.qualys')
# Create task for storage worker to store debug data in database (this
# task has no direct DB access due to scanners queue).
......@@ -170,7 +170,7 @@ def qualys_scan(self, url):
# {'errors': [{'message': 'Running at full capacity. Please try again later.'}], 'status': 'FAILURE'}
if data['errors'][0]['message'] == "Running at full capacity. Please try again later.":"Re-queued scan, qualys is at full capacity.")
raise self.retry(countdown=60, priorty=PRIO_NORMAL, max_retries=30, queue='scanners.qualys')
raise self.retry(countdown=60, priorty=PRIO_NORMAL, max_retries=100, queue='scanners.qualys')
# We have no clue, but we certainly don't want this running on the normal queue.
# The amount of retries has been lowered, as this is a situation we don't know yet, we don't have to
# keep on making the same mistakes at the API.
......@@ -178,7 +178,7 @@ def qualys_scan(self, url):"Too many concurrent assessments: Are you running multiple scans from the same IP? "
"Concurrent scans slowly lower the concurrency limit of 25 concurrent scans to zero. Slow down. "
"%s" % data['errors'][0]['message'])
raise self.retry(countdown=120, priorty=PRIO_NORMAL, max_retries=30, queue='scanners.qualys')
raise self.retry(countdown=120, priorty=PRIO_NORMAL, max_retries=100, queue='scanners.qualys')
log.exception("Unexpected error from API on %s: %s", url.url, str(data))
# We don't have to keep on failing... lowering the amount of retries.
......@@ -189,7 +189,7 @@ def qualys_scan(self, url):
20 to 25, simply because it matters very little when scans are ran parralel.
"""'Still waiting for Qualys result. Retrying task in 20 seconds.')'Still waiting for Qualys result. Retrying task in 15 seconds.')
# 10 minutes of retries... (20s seconds * 30 = 10 minutes)
# The 'retry' converts this task instance from a rate_limited into a
# scheduled task, so retrying tasks won't interfere with new tasks to be
......@@ -197,7 +197,7 @@ def qualys_scan(self, url):
# We use a different queue here as only initial requests count toward the rate limit set by Qualys.
# Do note: this really needs to be picked up within the first five minutes of starting the scan. If you don't
# a new scan is started on this url and you'll run into rate limiting problems.
raise self.retry(countdown=30, priorty=PRIO_HIGH, max_retries=30, queue='scanners')
raise self.retry(countdown=15, priorty=PRIO_HIGH, max_retries=100, queue='scanners')
......@@ -292,11 +292,12 @@ def service_provider_scan_via_api(domain):
# log.debug(vars(response)) # extreme debugging
log.debug("Running assessments: max: %s, current: %s, client: %s",
)"Assessments: max: %s, current: %s, this client: %s, this: %s",
return response.json()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment