Add a runbook for running PDMs manually using the toolbox pod (or using a Kubernertes Job)

Summary

Recently, there was a PDM which could not be run using the PDM pipeline that we run usually: This is because there is too much traffic and the lock that is required on a table to run this migration is hard to acquire (even during low-traffic APAC-EMEA transition time): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20640

This PDM was run manually by SSH-ing to the deploy VM and then running the migration class inside the Rails console. We can not use the Console VM for this use-case because the Console VM connects through pgBouncer, while for running migrations, we always need to connect directly to the primary DB. This is because the use of pgBouncer is not recommended when running long-running migrations:

When should you NOT use PgBouncer:

For use cases that require long-running operations, we recommend a direct connection on port 5432. For example:

• Schema changes and DDL

-- https://planetscale.com/docs/postgres/connecting/pgbouncer#when-should-you-not-use-pgbouncer

We can run the Rails console on the Toolbox pod and execute this migration. But there are some risks to that: The toolbox pod may be evicted from the node that it is running on, or it might be rescheduled to another node for some reason by Kubernetes. This is expected behavior in terms of Kubernetes, where a Pod is not "special". So, we should have a runbook which explains how to run PDMs manually when required.

Acceptance criteria

• Figure out whether using a Kubernetes Job is appropriate for this use-case
  • It is not a good idea in this case because creating the manifest for the Job will require multiple manual steps. It is better to use the existing Toolbox pod but to protect it from eviction
• Runbook to run PDMs manually if required

/blocks https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21368
/label ~kubernetes 
/relate https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20640