Improve CI response to rate limiting
Since the rollout of Cloud NAT to the private runner machines, we see the occasional rate limiting error from CI jobs when they upload artifacts. This is something out of the user's control, and would provide a bad user experience for customers if we were to roll out Cloud NAT to shared runners.
We do not enable application-level rate limiting on gitlab.com, so we avoid the problem described in gitlab#22465 (closed). We do enable rate limiting in haproxy.
We should investigate the application CI job lifecycle code to ensure it's respecting the rate limit reset headers sent by haproxy, and backing off appropriately. The frequency of observed errors could be due to a bug here.
Edited by Craig Furman