Increase wal-prefetch for archive replica

Production Change - Criticality 4 C4

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	postgres-archive
Change Team Members	@abrandl
Change Severity
Buddy check or tested in staging	Buddy
Schedule of the change	tbd
Duration of the change	5 min.

This came from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7724:

WAL prefetching for wal-e is set to 4 on the archive replica, to 32 on the delayed replica. The archive replica suffers from increased replication lag during high traffic times:

I would like to bump the WAL prefetching on the archive replica to 32, too. This is a manual change as recovery.conf isn't managed through Chef on these instances (this is not a patroni instance):

Config change

restore_command = '/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-fetch -p 32 "%f" "%p"'

Followed by a gitlab-ctl restart postgresql.

Validation

We should stop seeing increased replication lag after a while.

added C4 Database [PROJECT] ~7170326 labels

changed the description

LGTM!

Thanks @hphilipps !

I changed the setting and restarted postgres. The lag is now slowly trending downwards but we may need to wait a few hours to see if this is stabilizing or not.

https://prometheus-db.gprd.gitlab.net/graph?g0.range_input=30m&g0.expr=pg_replication_lag%7Benvironment%3D%22gprd%22%2Cfqdn%3D%22postgres-dr-archive-01-db-gprd.c.gitlab-production.internal%22%2Cinstance%3D%22postgres-dr-archive-01-db-gprd.c.gitlab-production.internal%3A9187%22%2Cjob%3D%22postgres%22%2Cstage%3D%22main%22%2Ctier%3D%22db%22%2Ctype%3D%22postgres-archive%22%7D&g0.tab=0

@abrandl The delay still is growing at the same rate. :-(

The wal-prefetching was bumped down to 12 as I recall experimenting with it a while ago and 32 was way over the top and slowing things down again.

I'll close here since the change is done.

closed

mentioned in issue #1184 (closed)