Skip to content

Support graceful shutdown of Workhorse connections

Stan Hu requested to merge sh-workhorse-graceful-shutdown into master

This commit does two things:

  1. Uses Go's graceful HTTP shutdown to close the listen port and wait for all HTTP handlers to finish. This will wait up to shutdown_timeout seconds, which is 0 by default, to preserve existing behavior.

  2. Kicks out all long poll queries by closing down the shutdown channel, which is monitored by the Redis key watcher.

#325114 (closed)

Testing

  1. Enable shutdown_timeout to be 60s.
  2. Run workhorse with -apiCiLongPollingDuration 120s.
  3. Run a register GitLab Runner. Verify it's sitting in long poll.
  4. Fake a slow endpoint by adding sleep 30 to a controller (e.g. app/controllers/admin/dashboard_controller.rb)
  5. Issue a SIGTERM or hit CTRL-C on Workhorse.
  6. Observe the log message says shutdown initiated.
  7. Observe that the /api/v4/jobs/request returns right away with a 204. Future requests fail.
  8. Observe that Workhorse will hang out for 30 seconds while the endpoint loads.
  9. Once the final request is loaded, Workhorse shuts down.
Edited by Stan Hu

Merge request reports