Zoekt nodes should stop an ongoing indexing request if they go over their critical watermark.
Problem
If a very large namespace is being indexed on a zoekt node, it can easily blow past the critical storage watermark and cause an incident.
References:
Proposal
Rails needs a way to send a SIGINT signal to zoekt node to stop all indexing.
Assuming a node that is doing a very long indexing request becomes offline because the indexing requests are less frequent:
On a scheduled interval, rails should attempt to find the disk utilization for offline nodes but are still reachable over http. For any node that is over the critical watermark configured in rails, it should send a SIGNINT signal to that node via http POST.
Edited by John Mason