Solidify CI::Workloads abstraction with AR model

What does this MR do and why?

Background

This work is expanding on the new Ci::Workloads::Workload model that is being introduced to support Duo Workflow and a couple of other use cases. The workload model was originally introduced in !176742 (merged) and the idea is that this model will allow us to run arbitrary workloads inside of a CI Pipeline on demand. For the time being it only abstracts the caller away from the ::Ci::CreatePipelineService and the details of constructing a valid .gitlab-ci.yml string.

The reason this abstraction exists is that we see demand for various ad-hoc workloads in many features in GitLab (see #328489 and gitlab-com/content-sites/handbook!10811 (merged) for more examples) and today it's easiest for us to build these features using the Ci::Pipeline as the execution environment. There are longer term architecture opportunities (e.g. step-runner#78) for us to have different ways to execute these arbitrary workloads. But having a lot of tight coupling in the codebase to pipelines and CI job yaml is going to make it hard to swap out this internal execution environment longer term. Aside from simply providing the ability to easily switch execution environments as our architecture changes it is also probably more maintainable to abstract developers away from all the complexity that comes with CI pipelines. The abstraction also helps with security as, for example, we are able to strip all unnecessary CI variables from workload jobs.

For reference we can look at !182840 for how this workload class may be used in more parts of the codebase in future. But there is no immediate need to migrate all those things to workloads just yet so instead we are focusing on the minimum needed to unblock Duo Workflow.

This MR

This MR makes some tweaks to the abstraction:

1. It introduces a WorkloadDefinition which is just a wrapper to prevent consumers from having to pass around a job Hash everywhere. Internally it just stores a Hash which describes a CI job
2. It introduces a SimplifiedWorkloadDefinitionBuilderService which is a builder class that allows consumers to set the various parts of the workload they need (e.g. image, commands, variables) without needing to know how to construct the CI YML hash. This class constructs the correct CI job definition from all the various settings provided
3. It updates the RunWorkloadService to pass in the newly created branch name to all workloads as CI_WORKLOAD_REF. Note that this is necessary in workloads because we skip passing most of the normal CI variables and that means we do not have CI_COMMIT_REF_NAME available in our jobs. Additionally this protects us from becoming coupled to the specifics of GitLab CI pipelines/jobs by depending less on the full set of predefined variables
4. Updates our Duo Workflow related code to use this new API

References

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. Setup Duo Workflow in your GDK following https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/duo_workflow.md

2. Enable the feature flags:

   Feature.enable(:duo_workflow)
   Feature.enable(:duo_workflow_in_ci)

3. Confirm workflows are working correctly in your GDK by running them from VS Code pointing at a project in your GDK

4. Add a docker runner to your GDK

5. [Optional] If you wish to see workflow execution logs in CI job logs, you can set debug: true in gdk config (gitlab.yml)

6. Get an api scoped personal access token and set a local env var $GDK_API_TOKEN

7. Start a workflow by calling the API, setting the project_id to a valid project id in your instance:

   curl -H "Private-Token: $GDK_API_TOKEN" -XPOST 'http://gdk.test:3000/api/v4/ai/duo_workflows/workflows?project_id=7&start_workflow=true&goal=Create+a+file+called+hello.sh+that+prints+hello.'

8. Check the builds page for the given project and look at the build logs

9. Download the artifacts associated with the job. It should contain a workflow.diff file that contains diff changes made by Duo Workflow.

10. For more detail on the workflow you can also open VS Code and look at the workflow output there

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #517411