Add MR pipeline support to AST CI/CD templates

What does this MR do and why?

Adds the AST_ENABLE_MR_PIPELINES environment variable to condition the MR pipelines support.

This change is made across all actively maintained AST CI/CD templates as stated in the issue's implementation plan: #410880 (closed)

This MR also includes a refactor of the specs for these CI/CD template which has been done in separate MR where further explanations are provided: !185149 (merged)

Warning

This change as a broad scope of inpact and should be monitored carefully.

What could go wrong?

CI template changes can impact customer's CI/CD configuration and sometimes prevent the whole pipeline to start.

How to monitor the change?

I've setup this dashboard to monitor the requests for our container images hosted on the gitlab.com/security-product registry: https://log.gprd.gitlab.net/app/r/s/7nSbk

We should also keep an eye on CI pipeline creation errors on gitlab.com: https://dashboards.gitlab.net/goto/HorKYW0Hg?orgId=1

We also have metrics on AST security scans but there is too much delay and we'll probably get customer feedback before we see these metrics impacted.

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

• #410880 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

⚠️ this MR updates several features. Testing and validating (exhaustively) this MR locally will be cumbersome.

To test these template you can either:

• run this branch in your gdk and use the include:template as usual
• use the include:remote and point to this refs

Here is an example project testing several scenarios: https://gitlab.com/gitlab-org/secure/tests/olivier/mr-pipelines-testing/example-with-updated-templates

Related to #410880 (closed)