Investigate/ease May 2023 cluster alerts for pubsub queueing

In May, we have gotten a series of alerts/s4 incidents with pubsub queueing on rails:

Incident list:

• production#14369 (closed)
• production#14134 (closed)

Scope of the issue:

• investigate, where possible the root cause of the backlogs - 1day
• check that retention for new AI indexes is appropriate.
• Scale down retention to 5 or 6 days for rails/gitaly logs and other logs to give us headroom.