Old Runner pods hanging around, eventually block new pods from starting

There seems to be some cases where old runner pods are never cleared. In our case, they seem to slowly build up until the maximum concurrency is reached, and then nothing starts anymore and we have to manually go in and kill them.

When you look at the pods, you see something like this:

Note that no pods should ever be running for 15 days, that is far longer than our timeout period.