Simulate database connection pool saturation (staging)

Context

This is part of the work to upgrade and migrate the GitLab.com container registry to a new version backed by a metadata database and online garbage collection (&5523 (closed)). This will be achieved following the gradual migration plan detailed in container-registry#374 (closed).

Before moving to production we will test a series of failure scenarios that were previously identified and documented in gitlab-com/runbooks!3628 (merged).

Scenario

Connection pool saturation.

Expectations

Impact

• API unable to serve requests;
• GC unable to process tasks.

Behavior

• API and/or GC fail to pull a connection from a pool at any given time;
• API requests will timeout with a 500 Internal Server Error response.

Observability

• Errors show up in Sentry and logs;
• Grafana dashboards reflect the magnitude of the impact on the API and pool saturation metrics.

Recovery

API and GC resume operations normally once the connection pool is no longer saturated.

Target Environment

This should be tested in staging as that is the only environment with a custom PostgreSQL cluster like we will find in production.