Profile Old and New Code Path for Pushes, Pulls, and Listing
Context
Before load testing of the registry in staging in migration mode gitlab#340634 (closed), gitlab#340635 (closed) it would be helpful to have some insight into the performance differences between the old (filesystem) and new (database) code path via the built-in pprof profiler.
In this issue, we'll perform some simple tests with a local registry using a local database and filesystem storage. While this does not mirror a realistic setup, it removes network delays from the data, so we can better understand the differences in the code itself.
Neither new nor old tests were run in migration mode.
Push
This test was conducted by using the seed.sh script pushing 30 images each with one tag.
While this script was running, a 30-second sample of profile data was collected from the pprof server of the registry via curl.
Old
New
Analysis
We lose a bit of performance on the new code path for manifest puts. This appears to partially be because we store the manifest configuration payload with the manifest in the database. Because of this, we need to retrieve the configuration payload during the put, which is a new extra step compared to the old code path.
Pull
This test was conducted by using the seed.sh script to push 30 images, each with five tags.
docker system prune -a --volumes was run to ensure a clean starting environment between runs.
After the images were seeded, and the local environment cleaned, a bash script was run in a loop to continuously pull images:
while true; do
for i in {0..29}; do
for j in {0..4}; do
docker pull localhost:5000/repository-${i}:tag-${j}
done
done
doneWhile this script was running, a 30-second sample of profile data was collected from the pprof server of the registry via curl.
Old
New
Analysis
We lose a bit of performance on the new code path for manifest pulls. It appears that it takes a bit longer to confirm that a blob is linked to the repository on the new code path compared to the old one.
Comparing results with the push test, it seems that repositoryStore.FindByPath seems to account for some of this slow down. It may be worth investigating single step repository scoped database queries for frequently accessed information, which would avoid a round trip to the database to confirm that a blob is linked in this case. It's worth investigating that we're not calling repositoryStore.FindByPath any more than we have to, as well, as repository scoped endpoints only need to make this call once.
List
This test was conducted by using the seed.sh script to push one image with 1100 tags.
After the image was tagged, a bash script was run in a loop to continuously list tags:
while true; do
curl localhost:5000/v2/repository-0/tags/list?n=2000
doneThe n=2000 keeps the database from paginating the tags, so there is a direct comparison, as the filesystem metadata does not support tag pagination.
Additionally, the new code path was tested with n=20 to observe the effects of pagination on performance.
Old
New (Full)
New (Paginated 20)
Analysis
Both old and new code paths performed similarly with this endpoint, for non-paginated calls. Paginated calls to the database were around 43% faster compared to the filesystem.