Balance zoekt nodes by query load
What does this MR do and why?
For traversal id searches, instead of using nodes.sample and getting a random node, we can now select a node with the least amount of load to perform a search. This load balancer tracks the weight of increasing load for project|group|global searches. All of this is behind the feature flag zoekt_load_balancer
This should help performance of zoekt webserver and avoid large spikes of search degradation when CPU is saturated from many expensive group searches happening concurrently.
AI Summary
This change adds intelligent load balancing to a search system that distributes queries across multiple search nodes. Previously, the system would randomly pick any available search node to handle queries, which could lead to some nodes being overloaded while others sit idle.
The new system tracks how busy each search node is by assigning different "weights" to different types of searches - project searches get a weight of 1, group searches get 3, and global searches get 5 (since they're more resource-intensive). When a search starts, the system increases the node's load counter, and when it finishes, the counter decreases.
The load balancer always picks the node with the lowest current load, ensuring work is distributed more evenly. It uses Redis (a fast in-memory database) to store and track these load counters across all nodes. The system also includes safety features like automatic cleanup of counters and error handling.
This improvement should result in better performance and more reliable search functionality, as no single search node will become overwhelmed while others remain underutilized. The changes include comprehensive tests to ensure the load balancing works correctly for different search scenarios.
References
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.