Benchmark concurrent git push operations
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
We have a customer doing stress testing (https://gitlab.zendesk.com/agent/tickets/78488) and testing the concurrency/speed of git pushes over HTTP.
They have a 3-node cluster with 8 cores, 32 GB RAM. They have played around with 5-25 unicorn workers, but no matter what they have seen that as the number of concurrent users rises, that the response times falls off and eventually the stress tester software fails with errors due to socket closures by the server.
The data they show is as follows:
| Concurrent reqs | Max ms | Avg ms | Min ms | Requests per minute |
|---|---|---|---|---|
| 5 | 4363 | 2835 | 2082 | 105.8 |
| 10 | 4494 | 2980 | 2195 | 201 |
| 25 | 10744 | 6784 | 2255 | 217 |
| 50 | 17521 | 7301.2 | 2643 | 410.9 |
| 75 | 9610 | 6784 | 2494 | 654 |
Watching htop, it doesn't look like the system is overutilized: CPU, RAM, and I/O seems to be low.
One issue is that a single git push takes about a second in 9.4. Looking at an strace dump of gitlab-workhorse, I see that this time is dominated by shelling out to the three different gitlab-shell hooks: pre-receive, update, and post-receive: gitlab-shell#97 (closed). Each of these takes about 200-300 ms to run due to loading dependencies (100-200 ms) and parsing YAML (~50 ms).
What we can't figure out is why adding more unicorn workers doesn't appear to improve the performance of handling pushes. I have a feeling this is due to fact that API requests get queued up for unicorn (e.g. each push requires several internal API endpoint calls), but it's possible something else is going on. We don't have enough data to know what's happening.
What I suggest is the following:
- Write a script that creates N number of projects (where N is the number of concurrent users) via the API.
- Make the script add a commit and push via HTTP in N different threads.
- Instrument Unicorn to see the average response and queue times
@joshlambert @pchojnacki @bjk-gitlab Is this something we can test this week?
/cc: @jimash