Skip to content

SE-3098: Use multiple threads per worker, remove deployment monitoring task.

Boros Gábor requested to merge threaded_workers_and_priorities into master

Created by: Kelketek

This pull request enables multiple threads per worker. This should solve the immediate problem of the workers being blocked without increasing memory usage or CPU in any significant sense, as all threads can share current process memory and our workloads are almost entirely IO-based. This takes us from an effective worker count of 3 to a count of 18.

To test this PR: Start up your local dev stack and try to provision a server. If the server provisions, it works. For contrast, if you were to do this on master, it would hang indefinitely as the single process would only have one thread, and a minimum of two workers is needed for provisioning.

A second test would be to verify that a new multi-appserver deployment runs smoothly and deactivates the old servers.

This PR also removes the reconfigure_load_balancers task, which was requested to be removed by @lgp171188 due to obsolescence.

This is a part 1 fix that will solve the immediate problem of not having enough workers to process the provisioning of servers. A set of follow-up PRs are coming which will handle the implementation of Huey priorities to further clear the technical debt surrounding this problem.

Merge request reports