Updating Gitaly Cluster E2E test logic

Resolves #238953 (closed)

Praefect used to elect a new primary node as soon as a healthy check failure occured on the primary node In the current version, Praefect doesn’t immediately elect a new primary node if the current one is unhealthy, and instead waits until it is necessary to serve a request while the current primary is unavailable, at which stage the election to choose a new primary occurs. For this change I've had to make a number of changes to how the flow of the test works to reflect these changes in logic, to help get this test to a state where we can consider removing it from quarantine.

Description of the test

gitlab-org/quality/testcases#976 (closed)

Check-list

• Confirm the test has a testcase: tag linking to an existing test case in the test case project.
• Note if the test is intended to run in specific scenarios. If a scenario is new, add a link to the MR that adds the new scenario.
• Follow the end-to-end tests style guide and best practices.
• Use the appropriate RSpec metadata tag(s).
• Ensure that a created resource is removed after test execution. A Group resource can be shared between multiple tests. Do not remove it unless it has a unique path. Note that we have a cleanup job that periodically removes groups under gitlab-qa-sandbox-group.
• Ensure that no transient bugs are hidden accidentally due to the usage of waits and reloads.
• Verify the tags to ensure it runs on the desired test environments.
• If this MR has a dependency on another MR, such as a GitLab QA MR, specify the order in which the MRs should be merged.
• (If applicable) Create a follow-up issue to document the special setup necessary to run the test: ISSUE_LINK
• If the test requires an admin's personal access token, ensure that the test passes on your local with and without the GITLAB_QA_ADMIN_ACCESS_TOKEN provided.