Throttle migration when WAL rate exceeds threshold

Extracted from #353395 (comment 892991285)

Overview

1. Indicator: Rate of generating WAL segments
2. Source: prometheus (see below)
3. Action: Pause or throttle when WAL rate exceeded a configured threshold
4. Parameter: rate threshold

The WAL rate corresponds to the write load on the cluster. Slowing down when a configured threshold is crossed helps to reduce load during peak write times.

This can be based on pg_xlog_position_bytes metric through prometheus.

Specifications

PromQL to get the WAL rate (for main): avg(postgres:pg_xlog_bytes_per_second:rate5m{env='gstg', type='patroni'})

Threshold set: 70 MB

Next Steps

• Db throttling has to be monitoring to see how it behaves for the set threshold.
• Also Iterate on WAL rate db health check indicator (#421694 - closed) aims to improve this metric.