Commit 63e9b6f0 authored by Sophie Brun's avatar Sophie Brun

New upstream version 1.0

parents
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
config.py
/config.py
facebook_user_details.py
generate_passwords.py
git_searcher.py
temptweets.txt
instaUsernameOsint.py
ip_to_neighboursites.py
test.py
test_domainOsint.py
testhtml.html
testreg.py
username_reddit.py
active_default_file_check.py
core/ui/migrations/*
*.swp
db.sqlite3
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/
# Translations
*.mo
*.pot
# Django stuff:
*.log
# Sphinx documentation
docs/_build/
# PyBuilder
target/
#Ipython Notebook
.ipynb_checkpoints
This diff is collapsed.
[![ToolsWatch Best Tools](https://www.toolswatch.org/badges/toptools/2016.svg)](https://www.toolswatch.org/2013/12/2013-top-security-tools-as-voted-by-toolswatch-org-readers/)
[![Black Hat Arsenal](https://www.toolswatch.org/badges/arsenal/2016.svg)](https://www.blackhat.com/us-16/arsenal.html#datasploit) US
[![Black Hat Arsenal](https://www.toolswatch.org/badges/arsenal/2016.svg)](https://www.blackhat.com/us-16/arsenal.html#datasploit) EU
# Overview of the tool:
* Performs OSINT on a domain / email / username / phone and find out information from different sources.
* Correlates and collaborate the results, show them in a consolidated manner.
* Tries to find out credentials, api-keys, tokens, subdomains, domain history, legacy portals, etc. related to the target.
* Use specific script / launch automated OSINT for consolidated data.
* Performs Active Scans on collected data.
* Generates HTML, JSON reports along with text files.
## Basic Usage:
```
____/ /____ _ / /_ ____ _ _____ ____ / /____ (_)/ /_
/ __ // __ `// __// __ `// ___// __ \ / // __ \ / // __/
/ /_/ // /_/ // /_ / /_/ /(__ )/ /_/ // // /_/ // // /_
\__,_/ \__,_/ \__/ \__,_//____// .___//_/ \____//_/ \__/
/_/
Open Source Assistant for #OSINT
website: www.datasploit.info
Usage: domainOsint.py [options]
Options:
-h, --help show this help message and exit
-d DOMAIN, --domain=DOMAIN Domain name against which automated Osint
is to be performed.
```
# Required Setup:
* Python 2.7 (because bunch of dependencies do not support Python 3.0)
* Bunch of python libraries (use requirements.txt)
## Detailed Tool Documentation:
> [http://datasploit.readthedocs.io/en/latest/](http://datasploit.readthedocs.io/en/latest/)
from __future__ import absolute_import
from . import username, emails, domain, ip, domainOsint, emailOsint, ipOsint, usernameOsint, datasploit
__all__ = ["username", "emails", "domain", "ip", "domainOsint", "emailOsint", "ipOsint", "usernameOsint", "datasploit"]
__title__ = 'datasploit'
__version__ = '1.0'
__author__ = 'Shubham Mittal, Sudhanshu Chauhan, Kunal Aggarwal'
__license__ = 'GPL-3.0'
del absolute_import
import requests
import re
import sys
list_urls = open("check_urls.txt")
existing_urls = []
host = sys.argv[1]
base_url = "http://" + host + "/"
print base_url
def check_page(url):
req = requests.get(url)
return req
#Checking non existig page
base_req = requests.get(base_url + "rejgwterlbjwfnvierwebjrwfebelivajr")
print "Setting base request code for non_existing page as " + str(base_req.status_code)
base_statuscode = base_req.status_code
#Check for any random non-existing-page
for read_from_file in list_urls:
pagetohit = read_from_file.strip("\n")
print "Checking %s" % (pagetohit)
if (check_page(base_url + pagetohit).status_code != base_statuscode):
existing_urls.append(base_url + pagetohit)
else:
pass
if (len(existing_urls) != 0):
print "\n[+] Testing done, following URLs are existing."
for foundpages in existing_urls:
print foundpages
print "\n"
print "Note: Different status_code were returned which means file exist. \nIn certain cases, application might be restricting file access by returning \n403 forbidden / Rate limiting which verifies that file exist.\n"
else:
"[-] No luck buddy..:("
import optparse
from domain_dnsrecords import fetch_dns_records,parse_dns_records
import requests
from termcolor import colored
parser = optparse.OptionParser()
parser.add_option('-e', '--email_file', action="store", dest="emailfile", help="File containing list of Email ids", default="spam")
parser.add_option('-s', '--subdomain_file', action="store", dest="subdomain_file", help="File containing list of subdomains.", default="spam")
class style:
BOLD = '\033[1m'
END = '\033[0m'
def run_active(filename,entity):
counter = 0
if entity == "subdomains":
hosts_with_http_or_https = []
might_be_vuln = []
subdomain_list = []
fh = open(filename, 'r')
for y in fh.readlines():
subdomain_list.append(y.strip("\n").strip("\r"))
print colored(style.BOLD + "\n[+] Running Active Scan on " + str(len(subdomain_list)) + " subdomains" + style.END, 'green')
print "\n"
for x in subdomain_list:
print x + ": ",
recrd = fetch_dns_records(x,"CNAME")
print recrd
if "No Records Found" not in recrd:
try:
req = requests.get("http://" + str(x), timeout=5)
print colored("[+] HTTP - " + str(x) + ":\t" + str(req.status_code), 'green')
#If response code is 404, might be a third party app without mapping
if req.status_code == 404 or req.status_code == 403:
might_be_vuln.append(["http", x, recrd, req.status_code])
hosts_with_http_or_https.append("http://%s" % x)
except:
pass
try:
req = requests.get("https://" + str(x), timeout=5)
print colored("[+] HTTPS - " + str(x) + ":\t" + str(req.status_code), 'green')
#If response code is 404, might be a third party app without mapping
if req.status_code == 404 or req.status_code == 403:
might_be_vuln.append(["http", x])
hosts_with_http_or_https.append("https://%s" % x)
except:
pass
else:
counter = counter + 1
print colored(style.BOLD + "\n[+] No CNAME record found for " + str(counter) + " subdomains \n" + style.END, 'green')
if len(might_be_vuln) != 0:
print "Following subdomains are affected by Subdomain Take Over Vulnerability\n"
for x in might_be_vuln:
print x
else:
print "No subdomains are affected by Subdomain Take Over Vulnerability\n"
elif entity == "emails":
print "Work in Progress"
options, args = parser.parse_args()
emailfile = options.emailfile
subdomain_file = options.subdomain_file
if emailfile != 'spam':
filename = emailfile
run_active(filename, "emails")
elif subdomain_file != 'spam':
filename = subdomain_file
run_active(filename, "subdomains")
else:
print 'Please pass filename'
<!DOCTYPE html>
<html>
<head>
<!--Import Google Icon Font-->
<link href="http://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
<!--Import materialize.css-->
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.98.1/css/materialize.min.css">
<!--Let browser know website is optimized for mobile-->
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<style>
body {
background-color: #f2f2f2;
}
.card {
max-height: 400px;
overflow-x: auto;
}
</style>
</head>
<body>
<div class="container">
<div class="row">
<div class="col s12">
<h2>DataSploit</h2>
<h5>Report for {DOMAIN_NAME}</h5>
</div>
</div>
<div class="row">
<div class="col s12 m9 l10">
{MAIN_HTML}
</div>
<div class="col hide-on-small-only m3 l2">
<div class="target" style="padding-top: 25px;">
<div style="font-size:16pt;"><strong>Navigation</strong></div>
<ul class="section table-of-contents" style="margin-top:0;">
{SCROLL_SECTION}
</ul>
</div>
</div>
</div>
</div>
<footer class="page-footer blue" style="padding-top:0">
<div class="footer-copyright">
<div class="container">
&copy; 2017 Shubham Mittal (DataSploit)
<a class="grey-text text-lighten-4 right" href="https://github.com/DataSploit/datasploit">View on Github</a>
</div>
</div>
</footer>
<!--Import jQuery before materialize.js-->
<script type="text/javascript" src="https://code.jquery.com/jquery-2.1.1.min.js"></script>
<!-- Compiled and minified JavaScript -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.98.1/js/materialize.min.js"></script>
<script>
$(document).ready(function(){
$('.scrollspy').scrollSpy({scrollOffset: 5});
$('.target').pushpin({});
});
</script>
</body>
</html>
web.config
robots.txt
htaccess.txt
trace.axd
readme.html
admin.php
admin
phpinfo.php
sitemap.xml
config.xml
crossdomain.xml
Joomla.xml
readme.txt
.git
admin/
wp-login.php
#Store all your config's here.
#added to gitignore so will not be syned
shodan_api=""
bing_api=""
github_access_token=""
builtwith_api=""
censysio_id=""
censysio_secret=""
facebook_access_token = ""
google_cse_key= ""
google_cse_cx = ""
flickr_api=""
hashes_api=""
instagram_api=""
instagram_secret=""
ipinfodb_api=""
jigsaw_api=""
jigsaw_password=""
jigsaw_username=""
linkedin_api=""
linkedin_secret=""
twitter_consumer_key=""
twitter_consumer_secret=""
twitter_access_token = ""
twiter_access_token_secret = ""
zoomeyeuser = ""
zoomeyepass = ""
clearbit_apikey = ""
emailhunter=""
jsonwhois=""
fullcontact_api = ""
mailboxlayer_api = ""
virustotal_public_api = ""
upgoingstar
nutanpanda
sudhanshu_c
kunalaggarwal92
\ No newline at end of file
#!/usr/bin/env python
import re
import sys
import optparse
import emailOsint
import domainOsint
import ipOsint
import usernameOsint
parser = optparse.OptionParser()
parser.add_option('-a', '--active', action="store", dest="domain", help="Launches Active Scans (work in progress)",
default="spam")
options, args = parser.parse_args()
def printart():
print "\t "
print "\t ____/ /____ _ / /_ ____ _ _____ ____ / /____ (_)/ /_"
print "\t / __ // __ `// __// __ `// ___// __ \ / // __ \ / // __/"
print "\t / /_/ // /_/ // /_ / /_/ /(__ )/ /_/ // // /_/ // // /_ "
print "\t \__,_/ \__,_/ \__/ \__,_//____// .___//_/ \____//_/ \__/ "
print "\t /_/ "
print "\t "
print "\t Open Source Assistant for #OSINT "
print "\t Website: www.datasploit.info "
print "\t "
def main(user_input):
printart()
print "User Input: %s" % user_input
if re.match('[^@]+@[^@]+\.[^@]+', user_input):
print "Looks like an EMAIL, running emailOsint...\n"
emailOsint.run(user_input)
elif re.match('^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$', user_input):
print "Looks like an IP, running ipOsint...\n"
ipOsint.run(user_input)
elif re.match('^[a-zA-Z\d-]{,63}(\.[a-zA-Z\d-]{,63}).$', user_input):
print "Looks like a DOMAIN, running domainOsint...\n"
domainOsint.run(user_input)
else:
print "Looks like a Username, running usernameOsint...\n"
usernameOsint.run(user_input)
if __name__ == "__main__":
try:
user_input = sys.argv[1]
except:
print "\n[-] Invalid Input. Exiting now..\n"
sys.exit(0)
main(user_input)
#!/usr/bin/env python
import os, subprocess, tempfile
def edit():
config_path = os.path.dirname(__file__)
config_file = "%s/config.py" % config_path
if not os.path.exists(config_file):
print "[+] Looks like a new setup, setting up the config file."
os.rename("%s/config_sample.py" % config_path, config_file)
fh = open(config_file)
config = fh.read()
fh.close()
f, fname = tempfile.mkstemp()
fh = open(fname, "w")
fh.write(config)
fh.close()
cmd = os.environ.get('EDITOR', 'vi') + ' ' + fname
subprocess.call(cmd, shell = True)
with open(fname, "r") as f:
config = f.read().strip()
fh = open(config_file, "w")
fh.write(config)
fh.close()
os.unlink(fname)
if __name__ == "__main__":
edit()
Datasploit allows you to perform OSINT on a domain_name, email_id, username and phoneNumber. In order to launch any script, lets first understand the nomenclature of these scripts:
* All the scripts meant to perform osint on domain starts with the keyword ***'domain_'***. Eg. domain_subdomains, domain_whois, etc. In similar manner, scripts for osint on email_id starts with ***'email_'***, eg. email_fullcontact.
* Scripts with an *underscore* are standalone scripts and collects data of one specific kind.
* Scripts without an underscore are the ones used for automated collection of data using standalone scripts. Eg. domainOsint.py
In order to run any script, pass the respective argument. For example, domainOsint and domain_subdomains.py will expect a domain name to be passed.
```
python domainOsint.py -d example.com
python domain_subdomains.py example.com
```
While, domainOsint will call all other domain_* scripts and list down data as well as dump the same in mongoDb, domain_subdomains and other such scripts will just list down data specific to their function.
domainOsint.py generates a JSON and an HTML report in reports folder as following hirarchy (example files are based on abcd.com domain)
../datasploit/reports
---------------------|------abcd.com
---------------------|------|------abcd.com_YYYY-MM-DD-HH-MM-SS.html
---------------------|------|------abcd.com_YYYY-MM-DD-HH-MM-SS.json
---------------------|------|------abcd.com_YYYY-MM-DD-HH-MM-SS.subdomains.txt
---------------------|------|------abcd.com_YYYY-MM-DD-HH-MM-SS.emails.txt
We need following API keys to run this tool efficiently:
shodan_api, censysio_id, censysio_secret, zoomeyeuser, zoomeyepass, clearbit_apikey, emailhunter, fullcontact, google_cse_key, google_cse_cx.
## Shodan_api
* [Register](https://account.shodan.io/register) an account in shodan.
* Visit your registered email id and activate the account.
* [Login](https://account.shodan.io/login) to your account and you will find the API keys under profile overview tab.
* Copy the API key and this is the value for *shodan_api* field in the config.py file.
## Censysio ID and Secret
* [Register](https://www.censys.io/register) an account in censysio.
* Visit your registered email id and activate the account.
* [Login](https://www.censys.io/login) to your account.
* Visit [Account](https://www.censys.io/account) tab to get API ID and Secret.
* Your API key is the value for *censysio_id* field and API Secret is the value for *censysio_secret* field in config.py file.
## Clearbit API
* [Register](https://dashboard.clearbit.com/signup) an account in clearbit.
* It will auto redirect to the account.
* Visit [API keys](https://dashboard.clearbit.com/keys) tab to get API key.
* Copy the API key and this is the value for *clearbit_apikey* field in the config.py file.
## Emailhunter API
* [Register](https://emailhunter.co/users/sign_up) an account in emailhunter.
* Click on activation link send to your registered email address and it will auto redirect to the account.
* Visit [API keys](https://emailhunter.co/api_keys) tab to get API key.
* Copy the API key and this is the value for *emailhunter* field in the config.py file.
## Fullcontact API
* [Register](https://portal.fullcontact.com/signup) an account in fullcontact.
* [Login](https://portal.fullcontact.com/signin/).
* It will ask for mobile number verification, complete that.
* You will be redirected to the page where you can get the API key.
* Additionally you will also get one email in the registered email id with API details.
* Copy the API key and this is the value for *fullcontact_api* field in the config.py file.
## Google Custom Search Engine API key and CX id
* Go to https://console.developers.google.com/ > Credentials
* Click on 'Create Credentials' and select API key.
* Click on restrict key.
* Select HTTP Headers (Websites) radio button.
* Add **.datasploit.info/\** in restrictions. This is done in order to stop unintentional usage of your api key.
* Copy the API key and click on save button. This is the value for *google_cse_key* field in the config.py file.
* Go to https://cse.google.com/cse/all, Click on Add button.
* In sites to search box, enter "pastebin.com" and "pastie.org"
* Give any name to your search engine and click on Create button.
* Go to https://cse.google.com/cse/all again and click on the search engine you just created.
* Click on the 'Search engine id' button and copy your search engine id. This is the value for *google_cse_cx* field in config.py file.
## Zoomeye Username and Password
* [Register](https://www.zoomeye.org/accounts/register) an user with zoomeye and use the credentials for this tool. (Don't worry if you are redirected to sso.telnet404.com. *This is how it works.)*
* Name of fields in the signup form - *1. email, 2. username, 3. nickname, 4. password, 5. confirm_password, 6. captcha*
* Once you fill out the details it will redirect you to the account page.
* There you will found something: *(Status: Inactive. Activate Now)*
* Click on activate now and two fileds will be populated.
* The first field will be captcha and the second one will be email id.
* Once you fill the email id in the second text box, click on send activation code.
* Check the activation code your email account.
* Put this activation code in the email id text box and click on determine.
* Now your account is activated and use those credentials in the tool.
* Email ID which you have used to sign up is your username and is the value for *zoomeyeuser* field in config.py
* Your account password is the value for *zoomeyepass* field in the config.py
Well, lets accept the fact that nothing goes well without contributors. Here is the list of people who have helped ([@datasploit](https://twitter.com/datasploit)) grow in its first phase.
##### Core Contributors:
Folks who took out time from busy schedule and got their hands dirty with the code:
* Shubham Mittal ([@upgoingstar](https://twitter.com/upgoingstar))
* Sudhanshu Chauhan ([@upgoingstar](https://twitter.com/sudhanshu_c))
* Kunal Aggarwal ([@KunalAggarwal92](https://twitter.com/KunalAggarwal92))
* Nutan Kumar Panda ([@nutankumarpanda](https://twitter.com/nutankumarpanda))
##### Mentors:
Chaps who were generous enough to give feedback and suggest changes:
* Anant Srivastata ([@anantshri](https://twitter.com/anantshri))
* Prashant Mahajan ([@prashant3535](https://twitter.com/prashant3535))
* Shadab Siddiqui ([@sh4ds1dd](https://twitter.com/sh4ds1dd))
##### Testers
Below people helped us by quickly adopting the tool and raised few naive issues we missed out:
* Sagar Belure ([@sagarbelure](https://twitter.com/sagarbelure))
* Chandrapal ([@bnchandrapal](https://twitter.com/bnchandrapal))
\ No newline at end of file
# Overview of the tool:
* Performs OSINT on a domain / email / username / phone and find out information from different sources.
* Correlates and collaborate the results, show them in a consolidated manner.
* Tries to find out credentials, api-keys, tokens, subdomains, domain history, legacy portals, etc. related to the target.
* Use specific script / launch automated OSINT for consolidated data.
* Available in both GUI and Console.
Following API configs are mandatory for proper results in domainOsint.py:
* shodan_api
* censysio_id
* censysio_secret
* zoomeyeuser
* zoomeyepass
* clearbit_apikey
* emailhunter
Other modules:
* github_access_token
* instagram_token
* instagram_client_id
* instagram_client_secret
* jsonwhois
## Before running the program, please make sure that you have:
* Changed the name of the file 'config_sample.py' to config.py
* Entered all the required APIs in config.py file, as mentioned above.
## Usage
To launch an automated OSINT on domain, shoot following query:
```
python domainOsint.py <domain_name>
```
You can also run an standalone script, e.g.you might want to only run the subdomain finding script and avoid all other modules. In such case, use below mentioned command. *All the files starting with domain_ requires a domain name to be passed as first argument. Same follows for email, ip, etc.*
```
python domain_subdomain.py <domain_name>
```
To launch an automated OSINT on domain, shoot following query:
```
python domainOsint.py <domain_name>
```
## SETUP and Contribution
* Change config_sample.py to config.py
```
mv config_sample.py config.py
```
* Configure respective API keys. Documentation for generating these keys will be shared very shortly. Believe us, we are working hard to get things in place.
* Sources for which API keys are missing, will be simply skipped for the search.
### Config files
### Python dependencies
```
pip install -r requirements.txt
```
If you have updated the code and want to push the pip dependencies in the requirements.txt
```
pip freeze > requirements.txt
```
## Overview
* Performs automated OSINT on a domain / email / username / phone and find out relevant information from different sources.
* Useful for Pen-testers, Cyber Investigators, Product companies, defensive security professionals, etc.
* Correlates and collaborate the results, show them in a consolidated manner.
* Tries to find out credentials, api-keys, tokens, subdomains, domain history, legacy portals, etc. related to the target.
* Available as single consolidating tool as well as standalone scripts.
* Performs Active Scans on collected data.
* Generates HTML, JSON reports along with text files.
## Why DataSploit???
Irrespective of whether you are attacking a target or defending one, you need to have a clear picture of the threat landscape before you get in. This is where DataSploit comes into the picture. Utilizing various Open Source Intelligence (OSINT) tools and techniques that we have found to be effective, DataSploit brings them all into one place, correlates the raw data captured and gives the user, all the relevant information about the domain / email / phone number / person, etc. It allows you to collect relevant information about a target which can expand your attack/defence surface very quickly. Sometimes it might even pluck the low hanging fruits for you without even touching the target and give you quick wins. Of course, a user can pick a single small job (which do not correlates obviously), or can pick up the parent search which will launch a bunch of queries, call other required scripts recursively, correlate the data and give you all juicy information in one go.
## Tool Background
Created using our beloved Python, DataSploit simply requires the bare minimum data (such as domain name, email ID, person name, etc.) before it goes out on a mining spree. Once the data is collected, firstly the noise is removed, after which data is correlated and after multiple iterations it is stored locally in a database which could be easily visualised on the UI provided. The sources that have been integrated are all hand picked and are known to be providing reliable information. We have used them previously during different offensive as well as defensive engagements and found them helpful.
## Setup
Worried about setup? Well, there are two major requirements here:
* Setting up the db, django, libraries, etc. We will soon have a script which will automate this for you, so can just go ahead and shoot the OSINT job.
* Feeding specific API keys for few specific sources. We are going to have a knowledge base where step by step instructions to generate these API keys will be documented. Sweet deal?
* [Click here to check step by step setup guide](/setupGuide/)
## Roadmap
Apart from this, in order to make it more useful in daily life of a pen-tester, we are working to make the tool as an extension of the other tools that pen-testers commonly use such as Burp Suite, Maltego etc. so that you can feel at home during the usage.
This page holds the setup guide you will need before kicking off the datasploit in your system. Please note that all the documentation is as per *nix machines, and the tool has not been thoroughly tested on Windows platform. If you would like to volunteer for the same, give us a shout at helpme@datasploit.info. Following are the quick steps to get you going:
If you want to work with web gui, follow the steps till 7. Otherwise, follow till 5th and you should be good to go.
### Step 1 - Download DataSploit to your system.
You can either use the git command line tools using the following command:
```
git clone https://github.com/datasploit/datasploit.git
```
, or you can simply download the zip file *([link](https://github.com/datasploit/datasploit/archive/master.zip))* and extract the same using unzip.
```
unzip master.zip
```
### Step 2: Install python dependencies
Go into the tool directory and install all the python libraries using the requirements.txt file. In case you encounter 'Permission Denied' error, use sudo.
```
cd master
pip install -r requirements.txt
```
### Step 3: Rename config_sample.py to config.py
Please make sure that config.py is added in your gitIgnore file so that this is not commited in any case. We care for your data too, and hence this tip. :)
```
mv config_sample.py config.py
```
### Step 4: Generate API Keys and paste inside config.py
Generate API keys using the *api Key Generation* guide at
> http://datasploit.readthedocs.io/en/latest/apiGeneration/
and enter the respective values in config.py file. Leave all other key value pairs blank.
Congratulations, you are now good to go. Lets go ahead and run our automated script for OSINT on a domain.