Gitlab runners are not picking up jobs

We have runners in digital ocean, they seem to be ok

but all of a sudden today jobs stopped getting picked up (we put everything under a ton of load as we were approaching a deadline)

1. We have restarted he runners
2. We have canceled all jobs and then attempted to run a single pipeline.
3. We have tried waiting

any ideas - nothing in the gitlab ui is helping.