Simulate high write load
Context
This is part of the work to upgrade and migrate the GitLab.com container registry to a new version backed by a metadata database and online garbage collection (&5523 (closed)). This will be achieved following the gradual migration plan detailed in container-registry#374 (closed).
During testing in staging, we should perform load tests against the registry and look for potential issues, such as database connection pool saturations and connection leaks. The outcome from this should, among other things, aid in defining appropriate connection pool settings for production.
We already performed write load tests while trying to provoke a database connection pool saturation in #336914 (closed). For this purpose, we have used Skopeo to sync images from DockerHub into our staging registry. This allowed us to perform realistic tests using authentic images but proved to be challenging to scale as images can be pretty large. Therefore the upload requests to the storage backend (GCS) take a large portion of the time, thus taking pressure off the server and the database.
Strategy
The intention is to go beyond what we did in #336914 (closed). For this purpose, we will generate random images with tiny (only a few KiB in size) layers and have them pushed to the registry.
Because the registry database is partitioned by top-level namespace, we need to create many top-level groups in GitLab and spread the load across them for a more realistic scenario.
In terms of requests per second (RPS), we intend to reach at least 10% of that from the production registry. Looking at the previous week, the average server RPS in production was 1110 (source).
Methodology
We have developed a series of scripts (source) that allowed us to create groups and projects in GitLab and execute the tests.
We have created 50 top-level groups and 100 projects in GitLab. The first 50 projects were spread evenly across groups, and the remaining were spread randomly.
We placed the test scripts (run.sh and run-docker-build-and-push.sh) in each of the client hosts and used them to perform the tests in "parallel" across hosts. We have used a different number of threads during the tests, which means that we had N'x'M' clients in total, where Nis the number of hosts andM` is the number of threads per host. Each thread is responsible for building and pushing images in a loop.
We have used a random number of layers per image, between 3 and 10, each with a random size between 1 and 10 KiB. Each image is built locally in each host and pushed to a random repository (among the 100) in the staging registry, all using Docker.
The following table shows the combination of configurations and the start and end timestamps for each test:
| Run | Number of client hosts | Number of client threads per host | Start | End |
|---|---|---|---|---|
| 1 | 6 | 1 | 2021-09-28 10:24 UTC | 2021-09-28 10:31 UTC |
| 2 | 6 | 3 | 2021-09-28 10:33 UTC | 2021-09-28 10:42 UTC |
| 3 | 6 | 8 | 2021-09-28 10:43 UTC | 2021-09-28 10:52 UTC |
| 4 | 6 | 32 | 2021-09-28 10:53 UTC | 2021-09-28 11:00 UTC |
| 5 | 12 | 16 | 2021-09-28 11:21 UTC | 2021-09-28 11:24 UTC |
Results
During run 4 I saw some uploads hanging on the clients, and I thought the hosts were bottlenecked by CPU and network bandwidth, so I decided to duplicate the hosts and halve the number of threads in run 5. However, after looking more carefully at metrics, I realized that the problem was not the clients but rather the server. It turns out that we have hit a K8s scaling limit and the load balancer "collapsed" during run 5. This was discussed in #340635 (comment 688900286), and a followup issue was raised (gitlab-com/gl-infra/delivery#2044 (closed)). The problem is unrelated to the application/database.
Apart from that, there were no issues, and the results are encouraging. A total of 4,965 images with 37,978 blobs were created and registered in the database during these tests.
Below we share screenshots of metrics. These were extracted from the various registry dashboards in Grafana, namely:
Application
There were only 3 application errors during all tests (source), all due to a 503 Service Unavailable from GCS (sample). So the error rate didn't move.
Regarding RPS, we have reached an API RPS that represents 17% of that in production.
The apdex dipped during the load peak as we couldn't efficiently provision more pods to accommodate the load (#340635 (comment 688900286)).
Both CPI and memory hovered at reasonable values and then retraced cleanly at the end of each test, which shows no significant leaks.
Database
We were close to 800 RPS on the database side with reasonable latencies.
We can also see that usage is quite pronounced for the repository_find_by_path and repository_create_or_find queries. We already identified a way to reduce this (container-registry#452 (closed)).
Connection Pool
Despite the increased load, the application-side connection pool saturation remained relatively low. The limit is currently set to 16 open connections per pod, but we never went above half of that. This means we can potentially use a similar or lower setting for the initial rollout in production.
Garbage Collector
No errors and low RPS. There are two workers per pod, and they run at a configurable interval (currently set to >= 5s +- 33% jitter).
We can see that the tasks started to accumulate during tests, and then they were dealt with. We can also see that the time between runs (reviews) was high because there were no tasks available before the tests (workers will back off exponentially in case there are no tasks), but as soon as they were detected, the delay between runs stayed as low as possible (5s +- 33% jitter) to process tasks ASAP.
Nothing odd about rate and latencies.
As expected, nothing was deleted during tests because we were uploading images, not deleting tags or manifests.
Service Aggregate
Load Balancer
Here we can see the load balancer "crash" during run 5.