Fix Kubernetes cluster health badge

What does this MR do and why?

This MR fixes the Kubernetes cluster health badge on the environments page. Before, we updated the health status to failed once one of the resources returned in a failed state. Now with the addition of the Kubernetes Watch API, we receive real-time updates and can see how the cluster state recovers. Due to this change, we now need to update the health status when all the resources recovered.

Screenshots or screen recordings

On the video, I create a new deployment, which results in a failing replicaSet. But after some time, the replicaSet recovers and shows up in a successful state. The health badge reflects these changes.

Screen_Recording_2023-11-23_at_16.00.33

How to set up and validate locally

1. Visit the Environments page:

   • Project -> Operate -> Environments if the new navigation is enabled
   • Project -> Deployments -> Environments if the new navigation is not enabled

2. Create/Edit an environment using the UI

3. Enable KAS on your GDK (steps 1-2 from the guide).

4. Visit the Project -> Infrastructure -> Kubernetes clusters page and create an agent following the instruction from the modal.

   • Select the "Connect a cluster" button
   • The modal should pop up
   • In the modal select "Select an agent or enter a name to create new"
   • You probably won't have any configured agents to show up in the list, create a new one by typing the name of your choice
   • The button should appear at the bottom of the list saying "Create agent: <your-agent-name>"
   • Select the button and click "Register" in the next view.
   • Save the token to use it in the next point.

5. Add the following configuration inside your project in .gitlab/agents/<your-agent-name>/config.yaml for the user_access agent:

   gitops:
     manifest_projects:
     - id: <your-group>/<your-project>
   user_access:
     access_as: 
       agent: {}
     projects:
     - id: <your-group>/<your-project-to-share-agent-with>

6. Note that the shared agents should be connected to the cluster in order to appear in the list. Please follow points 3-8 from the guide and then the (Optional) Deploy the GitLab Agent (agentk) with k3d section to create a local cluster and connect your agent with the cluster.

7. Select an agent from the dropdown in the Environments settings page and save the change.

8. Visit the Environments page and select the environment that has an associated agent.

9. Add/delete/restart pods or workload types in your cluster and verify the health badge changes in real-time.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #422945 (closed)