Simulate high read load
Context
We're upgrading and migrating the GitLab.com container registry to a new version backed by a metadata database and online garbage collection (&5523 (closed)), following the gradual migration plan detailed in container-registry#374 (closed).
During testing in staging, we should perform load tests and look for potential issues. The outcome from this should, among other things, aid in defining appropriate connection pool settings for production.
This issue is to perform isolated read tests. The GitLab.com registry API rate is +90% reads (source), so it is crucial to extend the previous tests from #340635 (closed) (writes and reads) focusing solely on reads.
Strategy
Go over the list of images (~5k) previously pushed to the staging registry in #340635 (closed) and validate each one. A test client picks a random repository from the list, lists its tags, and then loops over those tags and validates them. Validating a tag/image implies the following operations:
- Get the image manifest that the tag points to (1
GET); - Validate that the manifest is well-formed and its digest matches;
- Get the image configuration blob listed in the manifest and check that the digest matches (1
GET); - For each layer listed in the manifest, ensure the corresponding blob exists and the digest matches (N
HEAD/GET).
Apart from putting read load on the registry, this will also serve to check the validity/integrity of the images previously created in #340635 (closed).
Regarding requests per second (RPS), the intention is to go as high as possible, reaching at least 10% of that from the production registry. Looking at the previous week, the average API RPS in production was 1110 (source).
Methodology
We're going to use the crane CLI and most precisely, its validate command. This command validates an image as described above.
We'll start by listing the tags of a random repository (with crane ls) and then loop over those tags and call crane validate for each. Reads will be spread randomly across the 50 top-level groups and 100 projects previously created.
A series of scripts were created to conduct these tests (source). We placed these (run-validate.sh and run-repo-validate.sh) in each client host and used them to perform the tests in "parallel". Multiple runs were executed, with a different number of threads per client, which means that we have NxM clients in total, where N is the number of hosts and M is the number of threads per host. Each thread is responsible for (in an infinite loop) picking a random repository, listing tags, and validate each of them.
The following table shows the combination of configurations and the start and end timestamps for each test:
| Run | Number of client hosts | Number of client threads per host | Start | End | Notes |
|---|---|---|---|---|---|
| 1 | 6 | 1 | 2021-09-29 09:50 UTC | 2021-09-29 10:58 UTC | |
| 2 | 6 | 3 | 2021-09-29 10:00 UTC | 2021-09-29 10:08 UTC | |
| 3 | 6 | 8 | 2021-09-29 10:10 UTC | 2021-09-29 10:18 UTC | |
| 4 | 6 | 16 | 2021-09-29 10:20 UTC | 2021-09-29 10:28 UTC | At this point, we noticed that we were peaking in the K8s scaling limit and therefore decided not to go above this number of clients. |
| 5 | 6 | 16 | 2021-09-29 10:35 UTC | 2021-09-29 11:05 UTC | We tweaked the crane validate command options to not only HEAD the image layers but actually GET/download them and calculate their digest (by removing the --fast option). This is a more realistic scenario and provides another integrity validation. This variation requires downloading data from the storage backend, so a slightly lower server RPS is expected. We let it run for longer to make sure we see stability on longer executions. |
Results
If comparing these results with the ones for writes (#340635 (closed)), it's important to note that we doubled the maximum replicas per cluster from 10 to 20 before these tests (source). So we have doubled the scaling capacity, despite still using only 2 clusters out of 3 (gitlab-com/gl-infra/delivery#2044 (closed)).
A total of 60,014 image validations were performed during these tests, and all succeeded. We went as high as ~550 API RPS and ~1K database RPS. This API RPS represents ~50% of the production RPS at the time of writing! Overall, there were no application or database errors, and the results were very encouraging.
The only problem observed was that 2% of the tag list requests have failed. Looking at the registry access logs (source) we can see that there are no errors. Digging deeper, we can observe that these failures occurred randomly while crane was trying to authenticate against Rails (before speaking with the registry). This can be confirmed by looking at Workhorse logs (source). We've put a lot of (unusual) pressure on the Rails API during these tests as well, and the error percentage (and randomness) is too low to be a concern IMO.
Here are the full logs across all hosts for run 5: run-5-logs.zip
Below we share screenshots of metrics. These were extracted from the various registry dashboards in Grafana, namely:
Application
The server error rate didn't move during all tests. The apdex barely moved during all tests, but we can see an intriguingly dramatic drop exactly after stopping run 5. I'm unable to explain this. I don't see any indication that the registry became unresponsive or significantly degraded after stopping the tests, unlike what the drop suggests. I wonder whether this is a consequence of how I shut down the test clients and how that may or may not have affected the apdex calculation. Unlike on the write tests, all clients threads were programmed to stop at exactly the same time across all hosts, and no graceful shutdown was used.
In terms of RPS, we reached 550 RPS, ~50% of what we're seeing in production at the time of writing.
CPU and memory usage remained low, except for two specific pods that saw a significant spike in usage during the start of run 5. This might be a scaling artifact due to the sudden dramatic increase in demand after a retrace to nearly 0 RPS before starting run 5. The CPU and memory usage retrace was clean after each test run, which is great.
Database
Steadily cruising at 1k queries per second.
We can see pronounced usage for the repository_find_by_path and repository_create_or_find queries. We already identified a way to reduce this (container-registry#452 (closed)).
Connection Pool
We peaked at over 70 open connections to the database across the clusters. The number of in-use connections saw rapid swings, which is expected (otherwise, we would be leaking connections). Clean retrace to nearly zero after tests.
Distribution across pods was even, except for two pods, showing a similar pattern to that observed in CPU usage, which seems to corroborate the explanation for that.
The application-side connection pool saturation remained low across pods. Currently, the staging registry is configured to allow up to 16 connections per pod. This allowed us to reach ~50% of the production traffic with low pool saturation. This is an important takeaway for the production connection pool configuration.
Service Aggregate
Minor drop in apdex (due to the previously highlighted server apdex drop) and no movement on the error rate across all components.
The cluster saturation climbed to 100% during runs 4 and 5 (6 clients, 16 threads each). This means we provisioned as many replicas as we could (20 per cluster, 2 active clusters) to meet demand. CPU usage peaked at 80% during run 4 and remaining around 60% for round 5 (in this run, we downloaded blobs from the storage backend, which took some pressure off the server as we grabbed those directly from GCS).