Add new "stale" runner state

Background

In gitlab-com/gl-infra/production#5666 we have identified that many runners tend to go stale and are never removed from the database causing performance issues.

While discussing this solution @tmaczukin said:

Apart of one-time action of removing "old and inactive runners from GitLab.com" (which would need to be communicated way before to give users chance to review what is needed in their cases), maybe we should start thinking about an expiration date for inactive runners feature in GitLab?

Proposal

As an iterative step, we should mark the stale runners as such, for all runners that have not connected in over 3 months.

User facing changes would include:

• Add a new GraphQL API state for STALE state, for: a) runners that connected last 3 months ago. b) runners that never connected, but were created 3 months ago.
• Add a stale filter to the runners UI.
• Show a runner as stale badge in the list and view for a runner with a tooltip This runner last connected to GitLab more than 3 months ago, or never connected.

Out of scope

• We won't have a configurable expiration time: The first iteration of this can be 3-month "hardcoded" expiration time. Favoring convention over configuration.
• We won't yet have a "bulk delete" step for stale runners, this can be provided for users by providing APIs script that leverage the STALE state.