Can we rate-limit self-made API calls to gitlab.com

Summary

During the incident, gitlab.com generate lots of API traffic to itself. This resulted in so many jobs being generated,

Related Incident(s)

Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6587

Desired Outcome/Acceptance Criteria

Is it possible to add a rate limit that would protect us from so many calls from ourselves without blocking legitimate traffic when it's triggered. Can we identify traffic by user agent or similar and rate limit ourselves only? The outcome of this could be an MR to test this, or it could be a new issue defining the work specifically, or it could be to close this as in-feasible.

Associated Services

Corrective Action Issue Checklist

• Link the incident(s) this corrective action arose out of
• Give context for what problem this corrective action is trying to prevent from re-occurring
• Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
• Assign a priority (this will default to 'priority::4')