Backpressure mechanism for Zoekt indexing

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Background

When Zoekt nodes get full, the indexing queue grows rapidly. And on increasing the node count Sidekiq gets too busy to process all the pending jobs. Implement a backpressure mechanism to handle the indexing smoothly.

Proposal