README.md 8.57 KB
Newer Older
1
# Secure Vote Stress Test
Max Kaye's avatar
Max Kaye committed
2

Max Kaye's avatar
Max Kaye committed
3
4
5
This respository manages all docker, python, c, and haskell files for the Secure Vote Stress Test.

If you'd like the docs expanded please raise an issue.
Max Kaye's avatar
Max Kaye committed
6

Max Kaye's avatar
Max Kaye committed
7
8
The stress test is scheduled for 1488812400 (which is Mon, 06 Mar 2017 16:00:00 GMT or Tues, 7 Mar 2017 02:00:00 ADST)

9
10
11
12
13
## Requirements:

* You will need a new version of `docker-compose` and `docker`.
* You should also have 500GB of space to use (though you shouldn't _need_ more than 400 GB).
* This system expects Ubuntu 16.04 ONLY - no other OSs are supported yet.
14
* At _least_ 24 hrs setup time before the stress test as Bitcoin needs to sync with the network. It might be easier if you can copy your `.bitcoin` folder from elsewhere.
15
16
17
18
19
20
* At _least_ 1.5 MB/s down consistently

Tested with AWS and Digital Ocean nodes (though we are only using AWS atm as we found them more reliable performance wise).

## Recommended Setup

Max Kaye's avatar
Max Kaye committed
21
We test this on `c4.xlarge` and `m4.xlarge` machines on AWS. `t2.xlarge` underperformed by a large margin.
22
23
24

This is a very CPU intensive stress test.
Virtual servers will run hot all day, though some high performance desktops or laptops will be okay.
25

Max Kaye's avatar
Max Kaye committed
26
27
## I just want to run it!

28
29
**WARNING** - This will _not_ respect your existing docker containers. If you do want to run this locally run it in a VM! - **WARNING**

Max Kaye's avatar
Max Kaye committed
30
31
32
Requirements: have a remote server with Ubuntu 16.04 on it and the ability to SSH in (best via a key stored in `.ssh/id_rsa.pub`, etc).

* Clone the repository (https://gitlab.com/exo-one/svst-docker) to your machine.
33
34
* If you don't want to bootstrap the blockchain download: 
  * From `svst-docker/` run `./mainnet/0_setup_remote.sh <USER> <REMOTE>` from `./svst-docker` making sure to put in the 
Max Kaye's avatar
Max Kaye committed
35
  right username and hostname/IP. (Use bash on windows maybe if you're a windows user)
36
37
38
39
  * If you do want to bootstrap
    * First run `./mainnet/l1_setup_repos.sh USER REMOTE`
    * Then copy `.bitcoin` to `svst-docker/bitcoind-data/.bitcoin` as mentioned in the bootstrap section below
    * Then run `./mainnet/x_init.sh USER REMOTE` to start the containers
Max Kaye's avatar
Max Kaye committed
40
41
* This will SSH into a machine, install the requirements, build the containers, and start the stress test. 
  Tested on Ubuntu 16.10 and macOS 10.12.3
42

Max Kaye's avatar
Max Kaye committed
43
All scripts are intended to be run from the repository's root directory.
44
45

## Vote Explorer
Max Kaye's avatar
Max Kaye committed
46

47
48
49
50
51
52
We have a very basic vote-explorer dashboard available on port 8683 ("vote" on an old mobile keypad).
There are two main files: `index.html` and `debug.html`, you'll be directed to `index.html` by default.

## Basic Troubleshooting

If you get stuck try running `./2_refresh_all.sh` on the stress-test box which will redownload
Max Kaye's avatar
Max Kaye committed
53
54
all base images and rebuild all containers.

55
56
### How do _you_ run it?

57
58
We (at XO.1) will run it in 'producer' mode - part of this involves providing the secret key used to sign headers.
Headers are only signed during the stress test as an anti-DoS measure, SecureVote itself will not use signed headers
59
60
61
62
(this would introduce a central authority).

## Codebase Organisation

63
The intended structure _for the multiple repositories_ is:
64
65
66
67
68
69
70
71
72
73
74
75

```
└── svst-docker
    ├── bitcoin-nulldata
    ├── stress-test-pallet-verification
    ├── svst-docs
    ├── svst-haskell
    └── svst-python
```

This allows the docker setup to copy the local files into Dockerfiles.

Max Kaye's avatar
Max Kaye committed
76
77
## Base Images

78
There are some base images we build other containers from. The main benefit is a common setup and caching.
Max Kaye's avatar
Max Kaye committed
79
80
81
82

* `dev-base-1` - this is the general base image, meant to include as many dev resources as possible
* `dev-base-haskell` - this is the base image for Haskell utils, based on dev-base-1. It has an out of date version of `svst-haskell`, however, stack has been upgraded and most libraries have cached prebuilt versions, so it's much faster to build on top of (rather than recompiling the same files over and over)
* `dev-base-python` - this is like the haskell image above and also based on dev-base-1. Since python doesn't need to compile much and a pip install is fast it is just an interim layer for convenience
Max Kaye's avatar
Max Kaye committed
83
* `dev-test-bitcoind` - a testing image used to skip over compiling bitcoind; not to be used in production
Max Kaye's avatar
Max Kaye committed
84
85
86
87
88
89

## Microservice images

In the `dockerfiles` subdirectory there are a number of folders with corresponding docker files.
These are responsible for building and running each individual service.

90
91
There are also a number of dockerfiles in the main directory.

Max Kaye's avatar
Max Kaye committed
92
93
94
95
96
Our microservices are:

* `bitcoind` (Bitcoin-Nulldata)
* `ipfs` (decentralised content network)
* `postgres` (database, for data-base-y things)
97
* `producer` (for producing nulldata, headers, and pallets)
Max Kaye's avatar
Max Kaye committed
98
99
100
101
102
103
104
105
106
107
108
* `scraper` (for scraping nulldata)
* `header-download` (for downloading and verifying pallet headers)
* `pallet-download` (for downloading pallets from verified headers)
* `pallet-verifier` (for verifying the signatures of pallets and the pallets themselves)
* `state-maker` (for calculating the state - essentially running through all valid pallets to count votes)
* `vote-explorer` (a simple html interface to explore votes and show the state of the network)

These are all started through the `docker-compose.yaml` files

## Volumes

Max Kaye's avatar
Max Kaye committed
109
110
We use volumes to make the docker images able to be used through all stages of the apps lifetime. All _state_ should be
stored in these folders, essentially.
Max Kaye's avatar
Max Kaye committed
111
112
113

* `bitcoind` uses `./bitcoind-data` to store `.bitcoin/` including configuration files and the main blockchain
* `ipfs` uses `./ipfs-data` to store _all_ downloaded pallets and headers (except invalid ones)
Max Kaye's avatar
Max Kaye committed
114
115
  * keep in mind, all IPFS chunks are stored here and the other services only download them via the API temporarily to
  verify / process them.
Max Kaye's avatar
Max Kaye committed
116
117
* `postgres` uses `./postgres-data` to store the DB files

118
### Bootstrapping off existing Bitcoind data
Max Kaye's avatar
Max Kaye committed
119

120
121
122
If you have an existing bitcoin full node you can use it's blockchain data instead of downloading it from scratch (phew).
You will need to make sure `bitcoind` is shut down before copying any DB files, and you will want to make sure you don't
copy your wallet.dat if you use one.
123

124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
**WARNING** - You will need to delete your previous `bitcoin.conf`. The stress test will set up a new one with a known
RPC password automatically. Please also ensure you don't copy your wallet.dat over.

Copy `~/.bitcoin` to `svst-docker/bitcoind-data/.bitcoin` and it should be picked up
by the bitcoin docker container. You should do this before you bring the containers up.

Final directory structure should look like:

```
svst-docker/
├── bitcoind-data/
│   ├── bitcoind-data  # just an empty file
│   └── .bitcoin/
│       ├── bitcoin.conf
│       ├── blocks/
│       ├── chainstate/
        └── other files...
(more files)
```

144
145
146
147
148
149
150
151
152
153
154
155
156
157
### Building Individual Images

Upon occasion you will need to rebuild a single image after changes have been
made upstream.

You will need to supply docker-compose with the instruction to build, to not
use the existing cached image as well as which image needs rebuilding:

    % docker-compose build --no-cache image_name

ie: rebuilding scraper:

    % docker-compose build --no-cache scraper

158
159
160
You can use `_rebuild_ci_container.sh` if you'd like to rebuild on for the testing framework.
Alternatively you can also run `_reset_tests.sh`

161
## Testing
162
163
164
165

Testing done through `docker-compose`'s testing framework.

See `.gitlab-ci.yml` for latest testing entrypoint and setup (in `run-tests`)
Max Kaye's avatar
Max Kaye committed
166

167
You can run tests manually with `_run_tests.sh` or `_dev_run_test.sh container1 container2 ...` to rebuild only some specific containers.
Max Kaye's avatar
Max Kaye committed
168
169
170
171
172
173
174
175
176

If you need to rebuild all containers for CI run `_reset_tests.sh` - e.g. if you alter svst-haskell and need to re-compile

Use these scripts for various testing features:

* `_reset_tests.sh` - rebuild all test containers
* `_rebuild_ci_container.sh` - rebuild a single container (useful for testing a single container quickly)
* `_debug_btc` - run bitcoin-cli commands quickly
* `_cleanup_tests.sh` - stops all containers from CI
177

178
179
180
181
182
183
184
185
186
187
188
189
## Testnet

The testnet mode is operated through the `./testnet/` directory and mirrors the mainnet commands.

They're all designed to operate on remote machines.

Briefly these are the interesting commands:
* `./testnet/r0_initialize_remote.sh USER REMOTE` - this will just call `r1` and `x_testnet_init`
* `./testnet/r1_setup_repos.sh USER REMOTE`
* `./testnet/x_testnet_init.sh USER REMOTE`

If you're operating locally it's best to have a read of the above scripts, but then to use `r2_auditor.sh`
190
191
192
193
194

# Notes

As it turns out: the `.dockerignore` file is really important if you have any data in bitcoind-data or ipfs-data, etc.
If you don't have one your machine will transfer many 10s of GB to docker each build.
Max Kaye's avatar
Max Kaye committed
195