Fix flaky tests for ui_variable_non_inheritable_when_forward_pipeline_variables_false_spec

What does this MR do and why?

Addresses flakiness of the variable inheritance based pipelines spec by avoiding relying on page navigation and some improvements on how pipelines are fetched. Flakiness is likely due to the nature of this test (parent pipeline creating multiple children pipelines).

References

Relates to https://gitlab.com/gitlab-org/quality/test-failure-issues/-/work_items/35708

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #35708 (closed)