`check_interval` appears to be ignored when multiple pending jobs exist

Summary

When multiple pending jobs exist for execution, it appears that gitlab-runner will continue to try to fetch jobs until it runs out of pending jobs, reaches the runner limit, or reaches the global concurrent limit instead of respecting the check_interval for how often the runner should attempt to fetch jobs from the server. After some investigation, I saw that MR !198 (merged) added the check_interval parameter and it appears that MR !200 (merged) added commit fc88221b that appears to ignore the check_interval by immediately sending the runner back around to be processed instead of letting the check_interval expire first.

Steps to reproduce

In my system, I have 4 physical machines that each host gitlab-runner instances, each of which has a limit of 6 jobs and a check_interval of 30 seconds. In my sample scenario, I have 5 pending jobs in a pipeline to run.

Actual behavior

The behavior I see is that the first machine to request a job winds up iterating quickly and pulling down all 5 pending jobs without waiting on the check_interval, so the first machine to check for jobs takes all of the pending jobs and the other three machines wind up being idle, basically wasting their processing capacity and overloading the first machine. Effectively, load balancing winds up being broken because the work is not effectively spread among the waiting machines.

Expected behavior

I would expect that the first job would go to the first machine, then it would wait for 30 seconds before trying to fetch another job, so the second job would go to the second machine, etc. Once all four machines have pulled a job, then the 30 second check_interval would have expired for the first machine and it would pull a second job.

Relevant logs and/or screenshots

Oct 23 09:16:53 burr gitlab-runner[36005]: time="2018-10-23T09:16:53-05:00" level=info msg="Checking for jobs... received" job=82446 repo_url="https://git.starlab.io/crucible/lure.git" runner=cf4b02a0 #012<nil>
Oct 23 09:16:53 burr systemd[1]: Started Session c937 of user gitlab-runner.
Oct 23 09:16:53 burr systemd[1]: Started Session c938 of user gitlab-runner.
Oct 23 09:16:53 burr gitlab-runner[36005]: time="2018-10-23T09:16:53-05:00" level=info msg="Checking for jobs... received" job=82447 repo_url="https://git.starlab.io/crucible/lure.git" runner=cf4b02a0 #012<nil>
Oct 23 09:16:53 burr systemd[1]: Started Session c939 of user gitlab-runner.
Oct 23 09:16:53 burr systemd[1]: Started Session c940 of user gitlab-runner.
Oct 23 09:16:53 burr gitlab-runner[36005]: time="2018-10-23T09:16:53-05:00" level=info msg="Checking for jobs... received" job=82448 repo_url="https://git.starlab.io/crucible/lure.git" runner=cf4b02a0 #012<nil>
Oct 23 09:16:53 burr systemd[1]: Started Session c941 of user gitlab-runner.
Oct 23 09:16:53 burr systemd[1]: Started Session c942 of user gitlab-runner.
Oct 23 09:16:53 burr gitlab-runner[36005]: time="2018-10-23T09:16:53-05:00" level=info msg="Checking for jobs... received" job=82449 repo_url="https://git.starlab.io/crucible/lure.git" runner=cf4b02a0 #012<nil>
Oct 23 09:16:54 burr systemd[1]: Started Session c943 of user gitlab-runner.
Oct 23 09:16:54 burr systemd[1]: Started Session c944 of user gitlab-runner.
Oct 23 09:16:54 burr gitlab-runner[36005]: time="2018-10-23T09:16:54-05:00" level=info msg="Checking for jobs... received" job=82450 repo_url="https://git.starlab.io/crucible/lure.git" runner=cf4b02a0 #012<nil>

Environment description

This is a custom installation mainly using docker runners.

Used GitLab Runner version

Version:      9.5.1
Git revision: 96b34cc
Git branch:   9-5-stable
GO version:   go1.8.3
Built:        Wed, 04 Oct 2017 16:26:27 +0000
OS/Arch:      linux/amd64