4.1 Fetch the AI Catalog flow definition by reference in CI runs
What does this MR do and why?
This follows the proposal on the epic. Currently a custom AI Catalog flow goes into the CI job as DUO_WORKFLOW_FLOW_CONFIG, and the Duo CLI sends it back to DWS as an inline flow. I think DWS could fetch the flow from GitLab by workflow ID instead, in the same call that returns its agents.
With the ai_catalog_fetch_flow_config flag on, there are two new behaviours:
- The job gets
DUO_WORKFLOW_FETCH_FLOW_CONFIG=trueand no flow.Runtimes::Cisetsfetch_flow_configwhenWorkflow.fetch_catalog_flow_config?is true (a custom flow in a project with the flag on).StartWorkflowServicethen sets the marker in place of the flow. - Each connection from the workflow's own job sends a
flow_config_digestclaim. DWS checks the fetched flow against this claim. To get it,ConnectionConfigServicecallsGenerateTokenon DWS with the flow config, takes the claim from the token, and does not use the token. So each connection makes one extra call to DWS. #639172 looks at how to remove that call.
A resumed job follows the current flag state, so the job and the CLI agree if the flag changes between runs.
With the flag off, nothing changes. This used to return Catalog agents at preauthorization, but the pull approach doesn't need that, so I've replaced it.
The other parts are feat(catalog): 3.2d fetch the flow definition f... (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7746 - merged) for DWS and feat(cli): ask DWS to fetch the flow definition... (gitlab-org/editor-extensions/gitlab-lsp!4602 - merged) for the CLI. Workhorse also needs a newer generated Go client after the ai-assist change is merged. Let me know what you think!
Related to #634359 (closed).
How to set up and validate locally
This needs the DWS and CLI changes to run end to end. On its own, the specs cover both flag states:
bin/rspec ee/spec/models/ai/duo_workflows/workflow_spec.rb ee/spec/services/ai/duo_workflows/start_workflow_service_spec.rb ee/spec/services/ai/duo_workflows/runtimes/ci_spec.rb ee/spec/services/ai/duo_workflows/resume_workflow_service_spec.rb ee/spec/services/ai/duo_workflows/connection_config_service_spec.rb ee/spec/lib/gitlab/duo_workflow/client_spec.rb