Improve asynchronous pipeline processing

What does this MR do?

This MR improves asynchronous processing of pipeline.

Why was this MR needed?

It eliminates some race conditions and improves performance.

Does this MR meet the acceptance criteria?

• CHANGELOG entry added
• Tests
  • Added for this feature/bug
  • All builds are passing

What are the relevant issue / merge request numbers?

Related merge request: !6410 (merged)
Extracted from !6411 (closed)