[META] GitLab.com Latency

Successor to 'Make GitLab.com fast' https://gitlab.com/gitlab-com/infrastructure/issues/59

Performance improvement issues can be found in https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf8=%E2%9C%93&state=all&label_name[]=performance

Goals get the he p99 of the user facing latency https://performance.gitlab.net/dashboard/db/transaction-overview?panelId=2&fullscreen under 1s (we're now at 2s and details are in https://performance.gitlab.net/dashboard/db/daily-overview where you can expand the controller/sql/git/cache and the big problem are the git timings https://performance.gitlab.net/dashboard/db/daily-overview?panelId=14&fullscreen ) => this not showing requests using Gitaly

Frontend can also shave 1.7 seconds off https://gitlab.com/gitlab-org/gitlab-ce/issues/33391

1. Git acess: https://gitlab.com/gitlab-org/gitaly/ for git access ongoing as a project
2. Database team, "DB is under control, we only see spikes of load and locks which are related to builds triggered in CI, we need to change our polling to something that can actually scale, this will also allow us to remove the throttling that is preventing GitLab.com from going completely down)" We have multiple read replicas
3. Deprecate backend 4 https://gitlab.com/gitlab-com/infrastructure/issues/888
4. More unicorn processes per host https://gitlab.com/gitlab-com/infrastructure/issues/954
5. CI caching gitlab-org/gitlab-workhorse#85 (closed) https://gitlab.com/gitlab-org/gitlab-ce/issues/21698#note_19722633 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8084 ← need help testing, Jen-Shin Lin will help, backend on CI so that is a good fit. Plan 2 hours pair coding sessions.
6. GitLab.com latency monitoring page https://gitlab.com/gitlab-com/infrastructure/issues/948
7. Build and publish packages via CI gitlab-org/gitaly#39 (closed)
8. Go training https://dev.gitlab.org/gitlab/organization/issues/1051 (closed)
9. Continuous/canary deployments #1504 (closed)
10. Test running more app servers #1883 (closed)