README.md 12 KB
Newer Older
1
[![Code Climate](https://codeclimate.com/github/failmap/admin/badges/gpa.svg)](https://codeclimate.com/github/failmap/admin) [![Build Status](https://travis-ci.org/failmap/admin.svg?branch=master)](https://travis-ci.org/failmap/admin) [![Test Coverage](https://codeclimate.com/github/failmap/admin/badges/coverage.svg)](https://codeclimate.com/github/failmap/admin/coverage)
Johan Bloemberg's avatar
Johan Bloemberg committed
2

3 4 5 6 7 8 9 10 11 12 13
# Introduction
Failmap is a web application that continuously scans and evaluates (government) organization websites for security best practices and allows these results to be published on a map.

This repository contains the main application of Failmap, it consists of a public web frontend, a administrative interface and scanners.

To run a local testing/development instance see the instructions below.

For a public production installation please refer to: https://gitlab.com/failmap/server

![screenshot](docs/screenshot.png)

dev's avatar
dev committed
14 15
# Support fail map
We keep organizations on their toes to protect everyone's data. Do you like this? Your donation insures continuous support, updates,
Johan Bloemberg's avatar
Johan Bloemberg committed
16
and new features.
dev's avatar
dev committed
17 18 19 20 21 22 23 24 25 26 27 28 29 30

The Internet Cleanup Foundation helps cleaning up bad stuff on the web.

Donate to this project safely, easily and quickly by clicking on an amount below.

<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/5" target="_blank">&euro;5</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/10" target="_blank">&euro;10</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/25" target="_blank">&euro;20</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/50" target="_blank">&euro;50</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/100" target="_blank">&euro;100</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/200" target="_blank">&euro;200</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa/500" target="_blank">&euro;500</a>
<a href="https://useplink.com/payment/qaCyn8t6Tar7c5zVS6Fa" target="_blank">&euro;other</a>

31 32
# System requirements
Linux or MacOS capable of running Python3 and git.
Johan Bloemberg's avatar
Johan Bloemberg committed
33

34
# Software Requirements
dev's avatar
dev committed
35

36
Download and install below system requirements to get started:
dev's avatar
dev committed
37

38 39
- [git](https://git-scm.com/downloads) (download and install)
- [python3](https://www.python.org/downloads/) (download and install)
40 41 42
- Tox (`pip3 install --user tox`)
- [direnv](https://direnv.net/) (optional, download and install, then follow [setup instructions](https://direnv.net/), see Direnv section below)
- [Docker](https://docs.docker.com/engine/installation/) (optional, recommended, follow instructions to install.)
Johan Bloemberg's avatar
Johan Bloemberg committed
43

44
# Obtaining the software
Johan Bloemberg's avatar
Johan Bloemberg committed
45

46
In a directory of your choosing:
Johan Bloemberg's avatar
Johan Bloemberg committed
47

Elger Jonker's avatar
Elger Jonker committed
48
    # download the software
49
    git clone --recursive https://gitlab.com/failmap/admin/
Elger Jonker's avatar
Elger Jonker committed
50 51 52 53

    # enter the directory of the downloaded software
    cd admin

54 55
Use Direnv to manage environment (see Direnv section below). This manages the Python Virtualenv and `DEBUG` setting required for local development.

Elger Jonker's avatar
Elger Jonker committed
56
    direnv allow
Johan Bloemberg's avatar
Johan Bloemberg committed
57 58 59

# Quickstart

60 61
For the quickstart we assume the usage of Docker as it offers the most complete environment for development. This project aims to be environment agnostic and development without Docker is possible. See Development section below.

62 63
Below commands result in a failmap installation that is suitable for testing and development. It is
capable of handling thousands of urls and still be modestly responsive.
Johan Bloemberg's avatar
Johan Bloemberg committed
64

Elger Jonker's avatar
Elger Jonker committed
65 66
If you need a faster, more robust installation, please [contact us](mailto:hosting@internetcleanup.foundation).

67 68
Ensure Docker is installed and running. Execute the following command to bring up the environment:

Johan Bloemberg's avatar
Johan Bloemberg committed
69
    docker-compose up -d --build
70 71 72

This will build and start all required components and dependencies to run a complete instance of the Failmap project (grab some coffee the first time this takes a while).

Johan Bloemberg's avatar
Johan Bloemberg committed
73 74 75 76
The containers are run in the background, to view logs for all containers run:

    docker-compose logs -f

77 78
Notice: MySQL/Redis connection errors might be shown during startup. This is normal as the database container takes some time to startup. All related actions will be retried until they succeed. To check everything is correct see the `docker-compose ps` command below for the expected output.

79 80 81 82 83
You can now visit the [map website](http://127.0.0.1:8000/) and/or the
[admin website](http://127.0.0.1:8000/admin/) at http://127.0.0.1:8000 (credentials: admin:faalkaart).

The environment is aware of code changes in the `failmap_admin` folder. Services are automatically restarted to reflect the latest changes.

Johan Bloemberg's avatar
Johan Bloemberg committed
84
To stop the entire stack run: `docker-compose down`
85

86 87 88 89 90 91
There is a command-line application available to perform administrative tasks. To run it do:

    docker-compose exec admin failmap-admin

Further in this documentation the `failmap-admin` command is mentioned, when using the Docker environment always prepend `docker-compose exec admin` before the command.

92 93 94 95 96 97 98 99 100 101 102 103 104 105
To see all running components:

    $ docker-compose ps
          Name                    Command               State                 Ports
    ---------------------------------------------------------------------------------------------
    admin_admin_1      /usr/local/bin/autoreload  ...   Up       0.0.0.0:8000->8000/tcp
    admin_broker_1     docker-entrypoint.sh redis ...   Up       0.0.0.0:5672->5672/tcp, 6379/tcp
    admin_database_1   docker-entrypoint.sh mysqld      Up       0.0.0.0:3306->3306/tcp
    admin_loaddata_1   /usr/local/bin/failmap-adm ...   Exit 0
    admin_migrate_1    /usr/local/bin/failmap-adm ...   Exit 0
    admin_worker_1     /usr/local/bin/autoreload  ...   Up       8000/tcp

The platform consists of 2 external dependencies `broker` (redis), `database` (mysql) and 2 main components `admin` (web frontend and administrative environment), `worker` (async task executor).

106
Two tasks are run at startup `migrate` (database schema management) and `loaddata` (test and development data loading). The status of `Exit 0` indicated they have completed without issues.
107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131

Most composer commands can be run against individual components, eg:

    docker-compose logs -f worker

For more information consult docker composer [documentation](https://docs.docker.com/compose/) or:

    docker-compose -h

# Development

To perform non-Docker development, make sure all Requirements are installed. Run the following script to setup a development instance:

    tools/dev_setup.sh

After this run:

    # finally start the development server
    failmap-admin runserver

Now visit the [map website](http://127.0.0.1:8000/) and/or the
[admin website](http://127.0.0.1:8000/admin/) at http://127.0.0.1:8000 (credentials: admin:faalkaart).

The setup script performs the following steps:

Elger Jonker's avatar
Elger Jonker committed
132 133 134
    # download even more requirements needed to run this software
    pip3 install -e .

Elger Jonker's avatar
Elger Jonker committed
135 136 137
    # and download the development requirements
    pip3 install -r requirements.dev.txt

Elger Jonker's avatar
Elger Jonker committed
138 139 140 141
    # creates the database
    failmap-admin migrate

    # create a user to view the admin interface
142
    failmap-admin load-dataset development
Johan Bloemberg's avatar
Johan Bloemberg committed
143

Elger Jonker's avatar
Elger Jonker committed
144 145 146 147 148 149
    # loads a series of sample data into the database
    failmap-admin load-dataset testdata

    # calculate the scores that should be displayed on the map
    failmap-admin rebuild-ratings

150 151
# Scanning services (beta)

152 153 154 155 156 157 158 159 160 161 162 163 164
Todo: add celery beat information

Some scanners require RabbitMQ to be installed. We're currently in transition from running scanners
manually to supporting both manual scans and celery beat.

Each of the below commands requires their own command line window:

    # start rabbitmq
    rabbitmq-server

    # start a worker
    failmap-admin celery worker -ldebug

165 166 167
These services help fill the database with accurate up to date information. Run each one of them in
a separate command line window and keep them running.

Elger Jonker's avatar
Elger Jonker committed
168 169 170 171 172 173 174 175
    # handles all new urls with an initial (fast) scan
    failmap-admin onboard-service

    # slowly gets results from qualys
    failmap-admin scan-tls-qualys-service

    # makes many gigabytes of screenshots
    failmap-admin screenshot-service
176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216

# Using the software

## The map website

The website is the site intended for humans. There are some controls on the website, such as the
time slider, twitter links and the possibilities to inspect organizations by clicking on them.

Using the map website should be straightforward.

## The admin website

Use the admin website to perform standard [data-operations](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete),
run a series of actions on the data and read documentation of the internal workings of the failmap software.

The admin website is split up in four key parts:
1. Authentication and Authorization
This stores information about who can enter the admin interface and what they can do.

2. Map
Contains all information that is presented to normal humans.
This information is automatically filled based on the scans that have been performed over time.

3. Organizations
Lists of organizations, coordinates and internet adresses.

4. Scanners
Lists of endpoints and assorted scans on these endpoints.


# Troubleshooting getting started

If you need a specific branch, for example "mapwebsite"

    git checkout mapwebsite

This repository uses [submodules](https://git-scm.com/docs/git-submodule) to pull in
external dependencies. If you have not cloned the repository with `--recursive` or you need to
restore the submodules to the expected state run:

    git submodule update
217

218
# Development
219

220
## Code quality / Testing
221

Johan Bloemberg's avatar
Johan Bloemberg committed
222 223 224
This project sticks to default pycodestyle/pyflakes configuration to maintain code quality.

To run code quality checks and unit tests run:
225 226 227

    tox

228
To make life easier you can use `autopep8`/`isort` before running `tox` to automatically fix most style issues:
229

230
    tox -e autofix
231

232 233 234 235
To run only a specific test use:

    tox -- -k test_name

236
To only run a specific test suite user for example:
Johan Bloemberg's avatar
Johan Bloemberg committed
237

238
    .tox/py34/bin/failmap-admin test tests/test_smarturl.py
Johan Bloemberg's avatar
Johan Bloemberg committed
239

Johan Bloemberg's avatar
Johan Bloemberg committed
240 241 242 243 244
To generate coverage report after tests in HTML run:

    coverage html
    open htmlcov/index.html

Johan Bloemberg's avatar
Johan Bloemberg committed
245 246 247 248
Pytest allows to drop into Python debugger when a tests fails. To enable run:

    tox -- --pdb

249
## Direnv / Virtualenv
Johan Bloemberg's avatar
Johan Bloemberg committed
250

251 252
This project has [direnv](https://direnv.net/) configuration to automatically manage the Python
virtual environment. Install direnv and run `direnv allow` to enable.
Johan Bloemberg's avatar
Johan Bloemberg committed
253 254 255 256

Alternatively you can manually create a virtualenv using:

    virtualenv venv
257

Johan Bloemberg's avatar
Johan Bloemberg committed
258
Be sure to active the environment before starting development every time:
259

Johan Bloemberg's avatar
Johan Bloemberg committed
260
    . venv/bin/activate
261
    export DEBUG=1
262

263 264 265 266 267 268 269 270
# Known Issues

## Docker installation

### ERROR: for admin_database_1  Cannot start service database: Mounts denied:
As the error suggests, you're running the installation from a directory that is not shared with Docker. Change the docker configuration or run the installation from your user directory. You might receive this error if you run `docker-composer up` from /var/www/ or /srv/www/ as docker by default only has access to your user directory.


Johan Bloemberg's avatar
Johan Bloemberg committed
271 272 273 274 275 276 277 278 279 280 281 282
# Versioning

Version for the project is losely semver with no specific release schedule or meaning to version numbers (eg: stable/unstable).

Formal releases are created by creating a Git tag with the desired version number. These tags will trigger automated builds which will release the specified code under that version. Tags can be pushed from a local repository or created through the Gitlab interface: https://gitlab.com/failmap/admin/tags/new

Informal releases are created by new commits pushed/merged to the master. The version number of the last formal release will be suffixed with the current short Git SHA.

For all releases artifacts will be created. Currently only Docker containers are pushed into the [registry](https://gitlab.com/failmap/admin/container_registry). Each artifact will be tagged with the appropriate version (formal or informal). Where needed abstract tags will also be created/updated for these artifacts (eg: Docker build/staging/latest tags).

For local development informal release or a special `dev0` build release is used which indicates a different state from the formal releases.

283
# Thanks to
dev's avatar
dev committed
284
This project is being maintained by the [Internet Cleanup Foundation](https://internetcleanup.foundation).
285 286
Special thanks to the SIDN Fonds for believing in this method of improving privacy.

Johan Bloemberg's avatar
Johan Bloemberg committed
287
Thanks to the many authors contributing to open software.