Add hybrid approach polling/subscriptions

What does this MR do and why?

It was noted that relying on real-time pipelines for new pipelines creates an overflow of information on Build > pipelines. This approach uses polling every 10 seconds for new pipelines and relies on subscription for status updates to existing pipelines.

References

Screenshots or screen recordings

Before	After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #223264