Investigate more than usual job timeouts since May 17

There are a lot of examples, but I'm linking the ones I came across today:

date	job	branch	relevant data from job log	runner info
May 17, 2024 at 7:40:14 PM UTC	rspec unit pg14 29/32	master	see comment	Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd) on green-4.private.runners-manager.gitlab.com/gitlab.com/gitlab-org rpvz2FF9, system ID: s_d704414ba02a
May 19, 2024 at 1:49:55 AM UTC	rspec unit pg16 29/32	master	see comment	Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd) on green-4.private.runners-manager.gitlab.com/gitlab.com/gitlab-org rpvz2FF9, system ID: s_d704414ba02a
May 21, 2024 at 3:44:29 PM UTC	rspec unit pg14 20/32	cherry-pick-734e8efc-to-16-8 (high priority backport MR)	see comment	Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd) on green-1.private.runners-manager.gitlab.com/gitlab.com/gitlab-org 4bq1s9yM, system ID: s_9d15f3e4e7ae
May 21, 2024 at 4:34:08 PM UTC	rspec unit pg14 18/32	only-check-added-ff-files-with-danger	see comment	Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd) on green-3.private.runners-manager.gitlab.com/gitlab.com/gitlab-org QuQPoFsC, system ID: s_fc023362bf2f
May 21, 2024 at 7:23:17 PM UTC	rspec unit pg14 18/32	smh-reapply-tx-jobs	See comment	Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd) on green-4.private.runners-manager.gitlab.com/gitlab.com/gitlab-org rpvz2FF9, system ID: s_d704414ba02a
May 21, 2024 at 8:02:08 PM UTC	rspec unit pg14 18/32	j_lar/move_spec_to_ee (MR to fix priority1 master:broken )	see 2nd section of this comment	Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd)on green-2.private.runners-manager.gitlab.com/gitlab.com/gitlab-org GaSD-S1F, system ID: s_5651e5b5643b

Investigation

I have only managed to collect data at this point and perform preliminary data analysis. See Data Analysis section for findings so far