Limit the count of concurrent zoekt_index a node can index

Background

With the introduction of multi_index node assignments, a single node can have many pending indices simultaneously which is not ideal for the node assignment and might lead to over overprovisioned node.

Proposal

We should have a limit of concurrent pending/initializing namespaces for one node. If the assignment code detects that the node it wants to use is over the limit, we should skip the assignment. The limit of concurrent pending/initializing indices should be 1

Something like this:

diff --git a/ee/app/services/search/zoekt/namespace_assignment_service.rb b/ee/app/services/search/zoekt/namespace_assignment_service.rb
index 2dba4b317a14..74557fe2875e 100644
--- a/ee/app/services/search/zoekt/namespace_assignment_service.rb
+++ b/ee/app/services/search/zoekt/namespace_assignment_service.rb
@@ -60,6 +60,13 @@ def initialize_index_for_project(project)
           end
 
           node = Node.order_by_unclaimed_space.id_not_in(indices.map(&:zoekt_node_id)).last
+
+          # if the node has over the limit of pending/initializing, return early
+          if node.pending_or_initializing.count >= PENDING_OR_INITIALIZING_LIMIT
+            @early_return = true
+            return
+          end
+
           # early return if there is no node available
           if node.nil?
             @early_return = true