Reintroduce Duo Workflow Remote Execution in CI Pipelines

What does this MR do and why?

Duo Workflow is a new AI based feature of GitLab in which you can give an AI agent access to edit files, run commands and do other things within an isolated compute environment. At the moment the feature is only available (internally) in the VS Code extension. In that context it is spinning up a docker container and downloading the "Duo Workflow Executor" and running it inside the docker container. That executor is then responsible for communicating with the Duo Workflow Service over gRPC. The service gives instructions to edit files, run commands and make HTTP requests to GitLab Rails. These interactions account for all the various tokens needed. The executor->service communication uses a dedicated JWT token. Then there is a dedicated OAuth token used for executor->rails communication.

This MR reintroduces the ability to run Duo Workflows in CI Pipelines. We did have this functionality in the past but out of concerns for certain aspects of security and generally trying to focus on 1 way to use Duo Workflow we reverted. In the meantime we're trying to add this behaviour back and we've put it behind a separate feature flag so that we can develop this independently.

This MR implements the first step in gitlab-com/content-sites/handbook!10875 (closed) . It includes a Ci::Workloads::RunWorkloadService which was extracted from work in !176479 (closed) . That MR is using it for a different, but similar, reason. The idea of this new service is to be an abstraction around dynamically creating CI jobs to execute ad-hoc commands. The work is intended to be re-used at least in all the following places (and possibly more):

1. Running Workspaces in CI
2. On demand DAST scans
3. On-demand DAST validation
4. Container Scanning when a new container image is pushed to the repository
5. Slash commands for chatops

Likely the abstractions will be adapted in a subsequent MR as we try to work out the common functionality for these use cases.

References

1. gitlab-com/content-sites/handbook!10875 (closed)
2. https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/duo_workflow/#with-remote-ci-pipeline-execution

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to test

1. Setup Duo Workflow in your GDK following https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/duo_workflow.md

2. Enable the feature flags:

   Feature.enable(:duo_workflow)
   Feature.enable(:duo_workflow_in_ci)

3. Confirm workflows are working correctly in your GDK by running them from VS Code pointing at a project in your GDK

4. Add a docker runner to your GDK

5. [Optional] If you wish to see workflow execution logs in CI job logs, you can set debug: true in gdk config (gitlab.yml)

6. Get an api scoped personal access token and set a local env var $GDK_API_TOKEN

7. Start a workflow by calling the API, setting the project_id to a valid project id in your instance:

   curl -H "Private-Token: $GDK_API_TOKEN" -XPOST 'http://gdk.test:3000/api/v4/ai/duo_workflows/workflows?project_id=7&start_workflow=true&goal=Create+a+file+called+hello.sh+that+prints+hello.'

8. Check the builds page for the given project and look at the build logs

9. Download the artifacts associated with the job. It should contain a workflow.diff file that contains diff changes made by Duo Workflow.

10. For more detail on the workflow you can also open VS Code and look at the workflow output there

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Related to #511372 (closed)

Related to #499228 (closed)