Pipeline DAG processing technical debt

Description

We recently merged !30845 (merged) that introduces some amount of technical debt into our CI/CD Core Domain.

Some details can be found in !30845 (comment 338949428).

We should clean it up once we remove legacy processing and presumably legacy stages too. This is important because this affects our code CI/CD platform.

/cc @ayufan @fabiopitino @cheryl.li