The 50 GiB test flight experiment

Context

This is part of Test offline migration runtime with test registries of different sizes and storage backends epic that aims to perform tests against test registries of different sizes and using different storage backends, starting with GCS, S3, and the local storage filesystem.

Problem

While we have a matrix with several relevant combinations, we have not performed such tests at the scale proposed, where we use real environments and seed them on the fly with a lot of generated data.

Generating data at this scale can bring unforeseen performances of the testing setup.

Goal

This setup targets a smaller registry size to experiment and fine-tune first assumptions. That should leave us better prepared for bigger setups and forecast costs.

👣 Proposed Steps

• Use GitLab Environment Toolkit to set up a Reference Architecture of 1K (config examples), with the Container Registry enabled
• Use GCP container registry team account (HackyStack)
• Use eng-quality-ops-ci-cd-shared-infra for the AWS setup
• Seeder VM with a script pushes images until an estimated registry of 50 GB to the GitLab instance
• Run the offline migration tool
• Estimate running costs
• Teardown of 1k setup, seeder VM
• Lessons learned