Fix failing create_and_process_pipeline_spec

What does this MR do?

Refactor create_and_process_pipeline_spec to loop through each job individually rather than validate from the Pipeline page.
Navigate through each job, wait for it to be running, then validate that it's "passed", "failed" or "pending", which has its own built-in waiting.

Closes #202218 (closed)

Does this MR meet the acceptance criteria?

Conformity

• [-] Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• [-] Separation of EE specific content