Geo: Benchmark

Objectives

Measure throughput, understand latency / limitations.

Use minimum recommended specs:

• 2 Cores
• 2 GB RAM

(execute in Digital Ocean droplets with 2 GB/2 CPUs / 40 GB SSD)

Benchmark 1: notification channel throughput

Fire 10K hook notifications from Primary to Secondary in different geografical locations (NYC3 -> AMS3) (just receive an generate the jobs, workers must be turned off to isolate)

1. Open rails console on primary
2. Select a random project and emulate any notification trigger with something like : 10000.times { }

Based on that, decide if we should have dedicated "Geo" sidekiq queue (requires changes in omnibus) Evaluate which "order" or "priority" should it have (see: https://github.com/mperham/sidekiq/wiki/Advanced-Options)

Benchmark 2: concurrent updates

This is a little tricky to emulate, measure, but to have some data:

1. turn off secondary worker proccess
2. from a machine in the same availability zone as Primary
3. use dd to create 1Gb file, add that to 5 repositories in Primary.
4. wait for all the 5 pushes to settle on Primary
5. patch worker class to measure time it takes to proccess and output to log
6. start worker proccess, wait for execution to finish, and get results.

We could also use something like collectd to have some cpu/memory graphs to evaluate the load we put on the server to do that.