PipelineFailureManagement can fail in a non-graceful way when the API returns 500
I discovered we miss a few incidents/Slack notifications due to the API returning a 500 error: https://sentry.gitlab.net/gitlab/triage-ops/issues/4159010/?environment=production
We should rescue this and retry a few times before either:
- creating a new incident
- notifying about the API failure preventing from updating a potential existing incident