Testing (Async) registry BBM against Production like database

Context

The registry BBM process is an experimental feature that allows the registry to perform large scale data migrations in the container registry database: https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/gitlab/database-background-migrations.md

We need to test registry async Batched Background Migrations (BBMs) against a gitlab production like database to ensure that migrations are executed efficiently and safely, without negatively impacting system performance, user experience, or data integrity.

This will give us a higher level of confidence to release and benchmark real background in staging and production environments #1287 (closed):

1. Setup Test Environment

  • Prepare a controlled environment where testing/benchmarking can be performed (this could likely be a GitLab test instance with a container registry or just a stand alone container registry) with sufficiently populated database (database lab instance or mock data)

2. Define Test Migrations

  • Identify a batched background migrations (BBMs) you will benchmark. Choose different types of migrations, such as those with varying batch sizes, migration complexity and performance expectations.
  • Add the test migration(s) to the registry and build a registry containing the newly added migrations.

3. Run the background migrations

  • Use the built registry (from 2.) as your gitlab container registry for the gitlab environment created in 1.

  • Start/Execute the Background migrations

    Things to look out for:

    • Database Locking Issues (e.g Deadlocks)
      • Enable logging of database locks and deadlocks in PostgreSQL.
      • Monitor active sessions and queries in real-time to detect any contention caused by the migration.
    • Verify the database runs to completion and that data integrity is maintained
    • Keep an eye on system performance metrics (CPU, memory, I/O, query performance) as well as how long the a migration takes to run.
  1. Document, Analyze and share results, drawing special attention to any observed performance issues
  2. Verify that enforced bbm behave as expected (i.e. they stop the registry from starting if the enforcement was not complete prior)
Edited by Suleimi Ahmed