Sec Decomp testing in db-benchmarking
With gitlab-com/gl-infra/db-migration!566 (merged) merged, we can begin testing the database-side automation for Security Decomp (breaking physical replication, setting up forward/reverse logical replication, pausing/unpausing pgbouncer, etc...).
Ref: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/10334
Current status:
# Primary Cluster (01 is broken and needed to be reinitialized. This is going to take a bit to get back in sync).
root@patroni-sec-decomp-main-v16-01-db-db-benchmarking.c.gitlab-db-benchmarking.internal:~# gitlab-patronictl list
+ Cluster: db-benchmarking-patroni-sec-decomp-main-v16 (7432562132117319567) ---------+----------------+---------+------------------+----+-----------+---------------------+
| Member | Host | Role | State | TL | Lag in MB | Tags |
+-------------------------------------------------------------------------------------+----------------+---------+------------------+----+-----------+---------------------+
| patroni-sec-decomp-main-v16-01-db-db-benchmarking.c.gitlab-db-benchmarking.internal | 10.255.103.101 | Replica | creating replica | | unknown | |
+-------------------------------------------------------------------------------------+----------------+---------+------------------+----+-----------+---------------------+
| patroni-sec-decomp-main-v16-02-db-db-benchmarking.c.gitlab-db-benchmarking.internal | 10.255.103.102 | Replica | streaming | 4 | 14 | nofailover: true |
| | | | | | | noloadbalance: true |
+-------------------------------------------------------------------------------------+----------------+---------+------------------+----+-----------+---------------------+
| patroni-sec-decomp-main-v16-03-db-db-benchmarking.c.gitlab-db-benchmarking.internal | 10.255.103.103 | Leader | running | 4 | | |
+-------------------------------------------------------------------------------------+----------------+---------+------------------+----+-----------+---------------------+
# Standby Cluster
root@patroni-sec-decomp-sec-v16-01-db-db-benchmarking.c.gitlab-db-benchmarking.internal:~# gitlab-patronictl list
+ Cluster: db-benchmarking-patroni-sec-decomp-sec-v16 (7432562132117319567) ---------+----------------+----------------+---------------------+----+-----------+-----------------+----------------------------+---------------------+
| Member | Host | Role | State | TL | Lag in MB | Pending restart | Pending restart reason | Tags |
+------------------------------------------------------------------------------------+----------------+----------------+---------------------+----+-----------+-----------------+----------------------------+---------------------+
| patroni-sec-decomp-sec-v16-01-db-db-benchmarking.c.gitlab-db-benchmarking.internal | 10.255.104.101 | Replica | in archive recovery | 4 | 0 | * | max_connections: 1200->400 | |
+------------------------------------------------------------------------------------+----------------+----------------+---------------------+----+-----------+-----------------+----------------------------+---------------------+
| patroni-sec-decomp-sec-v16-02-db-db-benchmarking.c.gitlab-db-benchmarking.internal | 10.255.104.102 | Replica | in archive recovery | 4 | 0 | | | nofailover: true |
| | | | | | | | | noloadbalance: true |
+------------------------------------------------------------------------------------+----------------+----------------+---------------------+----+-----------+-----------------+----------------------------+---------------------+
| patroni-sec-decomp-sec-v16-03-db-db-benchmarking.c.gitlab-db-benchmarking.internal | 10.255.104.103 | Standby Leader | in archive recovery | 4 | | * | max_connections: 1200->400 | |
+------------------------------------------------------------------------------------+----------------+----------------+---------------------+----+-----------+-----------------+----------------------------+---------------------+
Edited by Jonathon Sisson