README.md 6.17 KB
Newer Older
Johan Bloemberg's avatar
Johan Bloemberg committed
1
[![Code Climate](https://codeclimate.com/github/failmap/admin/badges/gpa.svg)](https://codeclimate.com/github/failmap/admin) [![Build Status](https://travis-ci.org/failmap/admin.svg?branch=master)](https://travis-ci.org/failmap/admin) [![Test Coverage](https://codeclimate.com/github/failmap/admin/badges/coverage.svg)](https://codeclimate.com/github/failmap/admin/coverage)
Johan Bloemberg's avatar
Johan Bloemberg committed
2

dev's avatar
dev committed
3 4
# Support fail map
We keep organizations on their toes to protect everyone's data. Do you like this? Your donation insures continuous support, updates,
Johan Bloemberg's avatar
Johan Bloemberg committed
5
and new features.
dev's avatar
dev committed
6 7 8 9 10 11 12 13 14 15 16 17 18 19

The Internet Cleanup Foundation helps cleaning up bad stuff on the web.

Donate to this project safely, easily and quickly by clicking on an amount below.

<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/5" target="_blank">&euro;5</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/10" target="_blank">&euro;10</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/25" target="_blank">&euro;20</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/50" target="_blank">&euro;50</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/100" target="_blank">&euro;100</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/200" target="_blank">&euro;200</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/500" target="_blank">&euro;500</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa" target="_blank">&euro;other</a>

20 21
# System requirements
Linux or MacOS capable of running Python3 and git.
Johan Bloemberg's avatar
Docs.  
Johan Bloemberg committed
22

23
# Software Requirements
dev's avatar
dev committed
24

25
Download and install below system requirements to get started:
dev's avatar
dev committed
26

27 28
- [git](https://git-scm.com/downloads) (download and install)
- [python3](https://www.python.org/downloads/) (download and install)
29
- [direnv](https://direnv.net/) (download and install, the follow setup instructions)
dev's avatar
dev committed
30 31


32
After installation of above tools, all following steps use the command line:
Johan Bloemberg's avatar
Johan Bloemberg committed
33

34
    sudo easy_install pip  # installs pip, a python package manager, with the command pip3
Johan Bloemberg's avatar
Johan Bloemberg committed
35 36


37
# Obtaining the software
Johan Bloemberg's avatar
Johan Bloemberg committed
38

39
In a directory of your choosing:
Johan Bloemberg's avatar
Johan Bloemberg committed
40

41 42
    git clone --recursive https://github.com/failmap/admin/  # downloads the software
    cd admin  # enter the directory of the downloaded software
43
    direnv allow  # sets Debug to true in this folder. Do not change the settings.py file.
Johan Bloemberg's avatar
Docs.  
Johan Bloemberg committed
44 45 46

# Quickstart

47 48
Below commands result in a failmap installation that is suitable for testing and development. It is
capable of handling thousands of urls and still be modestly responsive.
Johan Bloemberg's avatar
Docs  
Johan Bloemberg committed
49

50
If you need a faster, more robust installation, please contact us.
Johan Bloemberg's avatar
Docs.  
Johan Bloemberg committed
51

52 53 54 55 56 57
    pip3 install -e .  # downloads requirements needed to run this software
    failmap-admin migrate  # creates the database
    failmap-admin createsuperuser  # create a user to view the admin interface
    failmap-admin load-dataset testdata  # loads a series of sample data into the database
    failmap-admin rebuild-ratings  # calculate the scores that should be displayed on the map
    failmap-admin runserver  # finally starts the server
dev's avatar
dev committed
58

59 60
Now visit the [map website](http://127.0.0.1:8000/) and/or the
[admin website](http://127.0.0.1:8000/admin/) at http://127.0.0.1:8000
Johan Bloemberg's avatar
Johan Bloemberg committed
61

62 63
# Scanning services (beta)

64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110
These services help fill the database with accurate up to date information. Run each one of them in
a separate command line window and keep them running.

    failmap-admin onboard-service  # handles all new urls with an initial (fast) scan
    failmap-admin scan-tls-qualys-service  # slowly gets results from qualys
    failmap-admin screenshot-service  # makes many gigabytes of screenshots

# Using the software

## The map website

The website is the site intended for humans. There are some controls on the website, such as the
time slider, twitter links and the possibilities to inspect organizations by clicking on them.

Using the map website should be straightforward.

## The admin website

Use the admin website to perform standard [data-operations](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete),
run a series of actions on the data and read documentation of the internal workings of the failmap software.

The admin website is split up in four key parts:
1. Authentication and Authorization
This stores information about who can enter the admin interface and what they can do.

2. Map
Contains all information that is presented to normal humans.
This information is automatically filled based on the scans that have been performed over time.

3. Organizations
Lists of organizations, coordinates and internet adresses.

4. Scanners
Lists of endpoints and assorted scans on these endpoints.


# Troubleshooting getting started

If you need a specific branch, for example "mapwebsite"

    git checkout mapwebsite

This repository uses [submodules](https://git-scm.com/docs/git-submodule) to pull in
external dependencies. If you have not cloned the repository with `--recursive` or you need to
restore the submodules to the expected state run:

    git submodule update
111

112
# Development
113

114
## Code quality / Testing
Johan Bloemberg's avatar
Johan Bloemberg committed
115

Johan Bloemberg's avatar
Johan Bloemberg committed
116 117 118
This project sticks to default pycodestyle/pyflakes configuration to maintain code quality.

To run code quality checks and unit tests run:
Johan Bloemberg's avatar
Johan Bloemberg committed
119 120 121

    tox

122
To make life easier you can use `autopep8`/`isort` before running `tox` to automatically fix most style issues:
Johan Bloemberg's avatar
Johan Bloemberg committed
123

124
    tox -e autofix
Johan Bloemberg's avatar
Johan Bloemberg committed
125

126 127 128 129
To run only a specific test use:

    tox -- -k test_name

130
To only run a specific test suite user for example:
Johan Bloemberg's avatar
Johan Bloemberg committed
131

132
    .tox/py34/bin/failmap-admin test tests/test_smarturl.py
Johan Bloemberg's avatar
Johan Bloemberg committed
133

Johan Bloemberg's avatar
Johan Bloemberg committed
134 135 136 137 138
To generate coverage report after tests in HTML run:

    coverage html
    open htmlcov/index.html

Johan Bloemberg's avatar
Johan Bloemberg committed
139 140 141 142
Pytest allows to drop into Python debugger when a tests fails. To enable run:

    tox -- --pdb

143
## Direnv / Virtualenv
Johan Bloemberg's avatar
Docs  
Johan Bloemberg committed
144

145 146
This project has [direnv](https://direnv.net/) configuration to automatically manage the Python
virtual environment. Install direnv and run `direnv allow` to enable.
Johan Bloemberg's avatar
Docs  
Johan Bloemberg committed
147 148 149 150

Alternatively you can manually create a virtualenv using:

    virtualenv venv
151

Johan Bloemberg's avatar
Docs  
Johan Bloemberg committed
152
Be sure to active the environment before starting development every time:
Johan Bloemberg's avatar
Johan Bloemberg committed
153

Johan Bloemberg's avatar
Docs  
Johan Bloemberg committed
154
    . venv/bin/activate
Johan Bloemberg's avatar
Johan Bloemberg committed
155
    export DEBUG=1
156

157
# Thanks to
dev's avatar
dev committed
158
This project is being maintained by the [Internet Cleanup Foundation](https://internetcleanup.foundation).
159 160
Special thanks to the SIDN Fonds for believing in this method of improving privacy.

Johan Bloemberg's avatar
Johan Bloemberg committed
161
Thanks to the many authors contributing to open software.