Add throttling mechanism to Elasticsearch migration backfill helper

What does this MR do?

This MR adds a throttling mechanism to the Elasticsearch migration backfill helper to prevent overwhelming the indexing queue during large migrations.

Why was this MR needed?

Elasticsearch migrations can produce a very large number of documents to index in a short period, which can overwhelm the Elasticsearch indexing queue and cause performance issues. This change adds an automatic throttling mechanism that pauses migrations when the queue size exceeds a threshold, allowing the system to catch up before continuing.

Technical details

1. Added a QUEUE_TRESHOLD constant set to 50,000 items
2. Added a check in the migrate method that pauses processing when queue is full
3. Added a queue_full? private method to check if bookkeeping service queue size exceeds threshold
4. Refactored service access through a bookkeeping_service method:
   • Makes the code more flexible for testing
   • Allows for potential extension in subclasses
5. Updated specs to test the throttling mechanism and service method

Testing

• Added tests to verify throttling behavior when queue is full
• Added tests for the bookkeeping service method
• Ran existing tests to ensure backward compatibility

Related issues

• Advanced Search: Pause backfill if queue is full (#542128 - closed)