Geo: Extend Unhealthy status threshold from 10 minutes to 1 hour

Michael Kozonorequested to merge
mk/extend-unhealthy-threshold into master

What does this MR do and why?

Partially mitigates #381354. A 10 minute old status is not a critical failure so the Geo site should not show a red "Unhealthy" in that case.

Customers with large sites may take 10 minutes to generate a status. In that scenario, the status will flap Unhealthy/Healthy.

A 1 hour old status is much more likely to indicate a problem.

So this MR increases the threshold from 10 minutes to 1 hour.

References

#381354

How to set up and validate locally

  1. Set up Geo https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/geo.md#easy-installation
  2. Visit Admin > Geo > Sites
  3. Wait for the secondary site status to be Healthy
  4. In the secondary GDK, gdk stop rails-background-jobs
  5. Wait 50 minutes
  6. Visit Admin > Geo > Sites
  7. The secondary site status should still be Healthy
  8. Wait 11 minutes
  9. The secondary site status should be Unhealthy

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michael Kozono

Merge request reports