WIP: Add retry limit and set it at four

What does this MR do?

This limits the amount of retries that the webhook worker can perform.

Are there points in the code the reviewer needs to double check?

Yes. I'm aiming at a 20 - 30 minute wait time for these retries based on feedback from the customer on the ticket link below which considers network timeout as the most common scenario where a retry is useful. Estimate is based on this formula (retry_count ** 4) + 15 + (rand(30) * (retry_count + 1))from the Sidekiq Erro Handling page.

Why was this MR needed?

Webhook workers commonly fail due to user misconfiguration or endpoint updates. Queues are then saturated with retries which make the entire instance unresponsive at some point. Limiting the number of retries to 4 instead of the default for 25 will help cleanup the queue.

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

• CHANGELOG entry added
• Documentation created/updated
• API support added
• Tests
  • Added for this feature/bug
  • All builds are passing
• Conform by the merge request performance guides
• Conform by the style guides
• Branch has no merge conflicts with master (if it does - rebase it please)
• Squashed related commits together

What are the relevant issue numbers?

• Bug report https://gitlab.com/gitlab-org/gitlab-ce/issues/23611
• Ticket ZD: https://gitlab.zendesk.com/agent/tickets/44858