Investigate disruptions during rollouts
During rollout operations there is a period in which the distributors and queries fail to send/query the ingesters.
During a rollout operation the rollout-operator updates 1 AZ at a time, and this should prevent any loss of quorum due to the AZ awareness.
However when 1 of the statefulsets is rotated (1 AZ) we see errors in the logs like removing ingester failing healthcheck.
Currently the ingesters are not set to deregister from the memberlist ring when shutdown, as this might possible cause the remaining ingesters to try to reshard the data.
We should investigate this and find the most optimal approach/setting to handle rollouts disruption free.
Edited by Nick Duff
