CI jobs are not evenly distributed among runners

Zendesk issue:

The customer describes that they have numerous Docker runners. Jobs seems to be distributed 100% to earlier workers before being sent to another runner. See the example below. Is this expected? Can we improve the distribution to better spread load across runner servers? I wasn't sure if this was a CI coordinator issue or runner so I created here.

No Build Que: 
Runner 1: (15 Concurrency) 
Runner 2: (15 Concurrency) 
Runner 3: (15 Concurrency) 
Runner 4: (15 Concurrency) 
Runner 5: (15 Concurrency) 
Runner 6: (15 Concurrency) 
Runner 7: (15 Concurrency) 
Runner 8: (15 Concurrency) 
Runner 9: (15 Concurrency) 
Runner 10: (15 Concurrency) 
Runner 11: (15 Concurrency) 
Runner 12: (15 Concurrency) 
Runner 13: (15 Concurrency) 
Runner 14: (15 Concurrency) 
Runner 15: (15 Concurrency) 
Runner 16: (15 Concurrency)

10 Jobs submitted: 
Runner 1: (15 Concurrency) 10 running 
Runner 2: (15 Concurrency) 
Runner 3: (15 Concurrency) 
Runner 4: (15 Concurrency) 
Runner 5: (15 Concurrency) 
Runner 6: (15 Concurrency) 
Runner 7: (15 Concurrency) 
Runner 8: (15 Concurrency) 
Runner 9: (15 Concurrency) 
Runner 10: (15 Concurrency) 
Runner 11: (15 Concurrency) 
Runner 12: (15 Concurrency) 
Runner 13: (15 Concurrency) 
Runner 14: (15 Concurrency) 
Runner 15: (15 Concurrency) 
Runner 16: (15 Concurrency)

Another 6 Jobs submitted: 
Runner 1: (15 Concurrency) 15 running 
Runner 2: (15 Concurrency) 1 running 
Runner 3: (15 Concurrency) 
Runner 4: (15 Concurrency) 
Runner 5: (15 Concurrency) 
Runner 6: (15 Concurrency) 
Runner 7: (15 Concurrency) 
Runner 8: (15 Concurrency) 
Runner 9: (15 Concurrency) 
Runner 10: (15 Concurrency) 
Runner 11: (15 Concurrency) 
Runner 12: (15 Concurrency) 
Runner 13: (15 Concurrency) 
Runner 14: (15 Concurrency) 
Runner 15: (15 Concurrency) 
Runner 16: (15 Concurrency)

cc/ @ayufan @tmaczukin

Edited Jul 15, 2020 by Jason Yavorsky
Assignee Loading
Time tracking Loading