Slack notifications not working for pipeline jobs - Handle rate limit events
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Work on this issue](https://contributors.gitlab.com/manage-issue?action=work&projectId=278964&issueIid=356896)
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=356896)
</details>
<!--IssueSummary end-->
I have added a slack integration to my project.
I am getting notifications for push events - however, pipeline jobs notifications are not working.
## Proposal
Some notifications are rate-limited, which we currently don't handle: https://gitlab.com/gitlab-org/gitlab/-/issues/356896#note_934257676
To fix this I think we would need to reschedule the failed `ProjectServiceWorker` job, using `retry_after` from the response (which might be minutes or seconds).
issue