ActiveContext: Lazy registration of configured queues

What does this MR do and why?

Problem

There is a static analysis issue in !199853 (comment 2665817994):

this is failing static analysis because, when Rails eager-loaded Ai::ActiveContext::Queues::Code, it also called the static methods .number_of_shards and .shard_limit, both of which in turn would call DB queries. The problem it seems is that there are no DB connections set up during Rails eager-loading yet

Unfortunately, we can't make these into instance methods because of the way ActiveContext::Concerns::Queue is setup, see: !199853 (comment 2663283992)

And !199853 (comment 2663283992):

Maybe we need to update ::ActiveContext::Concerns::Queue so it doesn't call #register! on include

Solution

In this change, we remove the base.register! call in the ActiveContext::Concerns::Queue, and instead do the following:

1. Introduce an ActiveContext config queue_classes
2. Introduce a new method in ActiveContext::Queues, register_all_queues! for registering all configured queues
3. When calling ActiveContext::Queues.raw_queues or ActiveContext::Queues.queues, register_all_queues! is called once
4. In the config/initializers/active_context.rb, we configure queue_classes with [::Ai::ActiveContext::Queues::Code]

Essentially, we are doing a lazy registration of the configured queues the first time either ActiveContext::Queues.raw_queues or ActiveContext::Queues.queues are called.

References

Related MR: ActiveContext: Allow flexible throughput adjust... (!199853 - merged)

Screenshots or screen recordings

N/A

How to set up and validate locally

1 - Update the number_of_shards in Ai::ActiveContext::Queues::Code

diff --git a/ee/lib/ai/active_context/queues/code.rb b/ee/lib/ai/active_context/queues/code.rb
index 1803f0b066e0..228e1a8d1b04 100644
--- a/ee/lib/ai/active_context/queues/code.rb
+++ b/ee/lib/ai/active_context/queues/code.rb
@@ -9,7 +9,7 @@ class Code
         class << self
           # having a single shard means we have absolute control over the amount of embeddings we generate in one go
           def number_of_shards
-            1
+            5
           end
         end
       end

2 - Check on the Rails console that ActiveContext is building the correct number of queues

ActiveContext::Queues.raw_queues
=> [#<Ai::ActiveContext::Queues::Code:0x00000001343e8818 @shard=0>,
 #<Ai::ActiveContext::Queues::Code:0x00000001343e85e8 @shard=1>,
 #<Ai::ActiveContext::Queues::Code:0x00000001343e8598 @shard=2>,
 #<Ai::ActiveContext::Queues::Code:0x00000001343e8570 @shard=3>,
 #<Ai::ActiveContext::Queues::Code:0x00000001343e8548 @shard=4>]

3 - Check that refs are still being queued correctly

refs_for_queue = (1..10).map(&:to_s)
::Ai::ActiveContext::Collections::Code.track_refs!(routing: "1", hashes: refs_for_queue)

ActiveContext::Queues.all_queued_items
=> {"ai_activecontext_queues:{code}:0:zset"=>
  ["Ai::ActiveContext::References::Code|2|1|3", "Ai::ActiveContext::References::Code|2|1|4", "Ai::ActiveContext::References::Code|2|1|10"],
 "ai_activecontext_queues:{code}:1:zset"=>
  ["Ai::ActiveContext::References::Code|2|1|5",
   "Ai::ActiveContext::References::Code|2|1|6",
   "Ai::ActiveContext::References::Code|2|1|7",
   "Ai::ActiveContext::References::Code|2|1|8"],
 "ai_activecontext_queues:{code}:2:zset"=>["Ai::ActiveContext::References::Code|2|1|1"],
 "ai_activecontext_queues:{code}:3:zset"=>["Ai::ActiveContext::References::Code|2|1|2"],
 "ai_activecontext_queues:{code}:4:zset"=>["Ai::ActiveContext::References::Code|2|1|9"]}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.