Run a test with more app servers?
Related to checking out performance #1830 (closed)
HTTP Queue timings sometimes spike to > 10s. https://performance.gitlab.net/dashboard/db/transaction-overview?panelId=13&fullscreen&orgId=1 (by the way, I can't find this on monitor.gitlab.net... why not?)
- download time series from this plot
- determine average, median, p95, p99
- consider adding twice the application servers for a week to see if it has an impact on these same stats; I would assume that the p99 and average would be reduced; but by how much depends on result of step 2.
Pablo has pointed out that the front end is not the main capacity hog at this point, these hosts are humming along, but the backend is just taking too long to answer the front end, if you remove the load you can see that 2 application servers can take the load and work twice as fast (staging). ==> This is true (staging twice as fast) but the issue here is specifically to see if it is worth addressing the peaks in http queue timings rather than the median.