Appropriate default delay for showing Geo node status stale indicator

Summary

With https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5653, we're now showing status icon next to Geo node stats when the status information is older than certain duration. However, this delay varies from environment to environment, for instance, GitLab instance operating at the scale of GitLab.com can give status information within 5 minutes of delay, while smaller installations have the same delay for up to a minute, so it is inappropriate to have a fixed, opinionated timeout coded into the application without considering the environment itself.

The following discussion from !5653 (merged) should be addressed:

• @kushalpandya started a discussion:

  @ash.mckenzie @stanhu Moving the discussion here to determine ideal default delay. 🙂

Proposal

Long term fix is to introduce a user configuration option in admin area where user can set desired timeout and application will then wait for the specified timeout before showing indicator on UI that the status information is stale. A short-term fix for now is to have a sweet-spot delay that works for us and any average size GitLab installation.