Several issues about 403s in runner PUT/PATCH endpoint
Drew asked me to create an issue about several of the issues we have encountered lately with 403's coming back from the PUT/PATCH endpoints. Some of them are related to the JWT token being enabled and not dropping the jobs on timeout but it appears there are other unrelated cases, since a few have been reported with the toggle off.
- Gitlab Runners and jobs get stuck due to a 403 ... (gitlab-runner#38356)
- Empty Shutdown() implementation prevents after_... (gitlab-org/fleeting/plugins/aws#98)
- https://gitlab.com/gitlab-com/request-for-help/-/issues/3692+
- https://gitlab.com/gitlab-com/request-for-help/-/issues/3716+
It should be noted that although there appears to be a correlation with using the AWS fleeting plugin in a few of the issues, the runner team doesn't think there is a runner side issue with the plugin.
I have identified the Issue with the JWT timeouts and created a draft MR here: !213394 (closed). I think someone else will need to pick this up and add specs. It should be very high priority since the JWT functionality can't be relied on with this bug.
cc. @drew, since you wanted me to compile these.