Webhooks may be rate limited and as a consequence not fire. We log the fact that a webhook has been rate limited in `app/services/web_hook_service.rb`, but it would be good to add alerting in the UI. We could adopt the approach used to address this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/363027) for adding alerts for disabled webhooks. This checks a redis cache which checks the `web_hooks` table to track whether a web hook is disabled. It uses the `user_namespace_callouts` DB table to track if an alert has been dismissed by a user. We could do something similar for rate limited webhooks. - add a column `last_rate_limited` column to the web_hooks table which is updated with a timestamp every time a web hook is rate limited. - store a true/false value in redis for each group and project that indicates whether the group/project has any recently rate limited webhooks. Recent could mean within the last hour. - display an alert if there is a recent rate limit. This won't display information about _which_ webhook, just that a webhook on the group/project has been rate limited and a link to the [webhooks troubleshoting page](user/project/integrations/webhooks_troubleshooting.md) - update the webhook API response to include the `last_rate_limited` timestamp. The implementation would involve additions to the following files: - `app/assets/javascripts/persistent_user_callout.js` - `app/assets/javascripts/persistent_user_callouts.js` - `app/models/concerns/web_hooks/has_web_hooks.rb` - `app/models/concerns/web_hooks/hook.rb` - `app/helpers/web_hooks/web_hooks_helper.rb` - `ee/app/helpers/ee/web_hooks/web_hooks_helper.rb` - `app/helpers/users/callouts_helper.rb` - `app/services/web_hook_service.rb` - `app/services/web_hooks/log_execution_service.rb` - `app/views/layouts/project.html.haml` - `app/views/layouts/group.html.haml` - `app/views/shared/web_hooks/_web_hook_rate_limited_alert.html.haml` (a new file) - `ee/app/views/shared/web_hooks/_group_web_hook_rate_limited_alert.html.haml` (a new file) Plus a DB migration to add the `last_rate_limited` column to `web_hooks`. Plus updating the API response endpoints. - `ee/spec/fixtures/api/schemas/public_api/v4/group_hook.json` - `lib/api/entities/hook.rb` It should be split into multiple MRs. We should be aware that at the time of creating this issue around 2.5K webhooks are rate limited each week on gitlab.com.