Support graceful shutdown of Workhorse connections
This commit does two things:
-
Uses Go's graceful HTTP shutdown to close the listen port and wait for all HTTP handlers to finish. This will wait up to
shutdown_timeout
seconds, which is 0 by default, to preserve existing behavior. -
Kicks out all long poll queries by closing down the shutdown channel, which is monitored by the Redis key watcher.
Testing
- Enable
shutdown_timeout
to be60s
. - Run workhorse with
-apiCiLongPollingDuration 120s
. - Run a register GitLab Runner. Verify it's sitting in long poll.
- Fake a slow endpoint by adding
sleep 30
to a controller (e.g.app/controllers/admin/dashboard_controller.rb
) - Issue a SIGTERM or hit CTRL-C on Workhorse.
- Observe the log message says
shutdown initiated
. - Observe that the
/api/v4/jobs/request
returns right away with a 204. Future requests fail. - Observe that Workhorse will hang out for 30 seconds while the endpoint loads.
- Once the final request is loaded, Workhorse shuts down.
Edited by Stan Hu