GitLab Recon

In CE to make our security product more popular.

The ideal flow is: enter a domain name 'example.com' and get a list of vulnerabilities with the most likely ones on top (sorted by signal to noise ratio).

To do this there are two components, Recon (this issue) and DAST https://gitlab.com/gitlab-org/gitlab-ee/issues/3956

The first iteration user flow will be:

1. Create a new project with the 'Recon' GitLab CI template.
2. Enter the TARGET_DOMAIN environmental variable.
3. Receive a list of subdomains in a yaml compatible format.

Rest of the workflow, not part of Recon:

1. Copy (part of) that list to the clipboard
2. Create a new project with the 'DAST' GitLab CI template.
3. Past the list into the .gitlab-ci.yml file to run the DAST in parallel against all subdomains.
4. Get a list of vulnerabilities with the most likely ones on top

The GitLab Recon CI template has two stages:

1. Running a number of subdomain discovery tools in parallel.
2. Running a task that aggregates their output and deduplicates it.

The subdomain discovery tools would be based on https://news.ycombinator.com/item?id=15676951 Example commands:

• /cdx-index-client.py -c CC-MAIN-2017-43 '*.example.com' with http://index.commoncrawl.org/
• python sublist3r.py -d example.com with https://github.com/aboul3la/Sublist3r
• ./ct.py example.com | ./bin/massdns -r resolvers.txt -t A -q -a -o -w with https://github.com/appsecco/bugcrowd-levelup-subdomain-enumeration
• python dnsrecon.py -n ns1.example.com -d example.com -D subdomains-top1mil-5000.txt -t brt with https://github.com/darkoperator/dnsrecon
• `dig +multi AXFR @ns1.exaple.com example.com
• ldns-walk @ns1.example.com example.com with https://www.nlnetlabs.nl/projects/ldns/

/cc @kathyw @dzaporozhets @bikebilly @markpundsack