Support graceful shutdown of Workhorse connections
This commit does two things:
-
Uses Go's graceful HTTP shutdown to close the listen port and wait for all HTTP handlers to finish. This will wait up to
shutdown_timeoutseconds, which is 0 by default, to preserve existing behavior. -
Kicks out all long poll queries by closing down the shutdown channel, which is monitored by the Redis key watcher.
Testing
- Enable
shutdown_timeoutto be60s. - Run workhorse with
-apiCiLongPollingDuration 120s. - Run a register GitLab Runner. Verify it's sitting in long poll.
- Fake a slow endpoint by adding
sleep 30to a controller (e.g.app/controllers/admin/dashboard_controller.rb) - Issue a SIGTERM or hit CTRL-C on Workhorse.
- Observe the log message says
shutdown initiated. - Observe that the
/api/v4/jobs/requestreturns right away with a 204. Future requests fail. - Observe that Workhorse will hang out for 30 seconds while the endpoint loads.
- Once the final request is loaded, Workhorse shuts down.
Edited by Stan Hu