4.1 Fetch the AI Catalog flow definition by reference in CI runs

What does this MR do and why?

This follows the proposal on the epic. Currently a custom AI Catalog flow goes into the CI job as DUO_WORKFLOW_FLOW_CONFIG, and the Duo CLI sends it back to DWS as an inline flow. I think DWS could fetch the flow from GitLab by workflow ID instead, in the same call that returns its agents.

With the ai_catalog_fetch_flow_config flag on, there are two new behaviours:

  • The job gets DUO_WORKFLOW_FETCH_FLOW_CONFIG=true and no flow. Runtimes::Ci sets fetch_flow_config when Workflow.fetch_catalog_flow_config? is true (a custom flow in a project with the flag on). StartWorkflowService then sets the marker in place of the flow.
  • Each connection from the workflow's own job sends a flow_config_digest claim. DWS checks the fetched flow against this claim. To get it, ConnectionConfigService calls GenerateToken on DWS with the flow config, takes the claim from the token, and does not use the token. So each connection makes one extra call to DWS. #639172 looks at how to remove that call.

A resumed job follows the current flag state, so the job and the CLI agree if the flag changes between runs.

With the flag off, nothing changes. This used to return Catalog agents at preauthorization, but the pull approach doesn't need that, so I've replaced it.

The other parts are feat(catalog): 3.2d fetch the flow definition f... (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7746 - merged) for DWS and feat(cli): ask DWS to fetch the flow definition... (gitlab-org/editor-extensions/gitlab-lsp!4602 - merged) for the CLI. Workhorse also needs a newer generated Go client after the ai-assist change is merged. Let me know what you think!

Related to #634359 (closed).

How to set up and validate locally

This needs the DWS and CLI changes to run end to end. On its own, the specs cover both flag states:

bin/rspec ee/spec/models/ai/duo_workflows/workflow_spec.rb ee/spec/services/ai/duo_workflows/start_workflow_service_spec.rb ee/spec/services/ai/duo_workflows/runtimes/ci_spec.rb ee/spec/services/ai/duo_workflows/resume_workflow_service_spec.rb ee/spec/services/ai/duo_workflows/connection_config_service_spec.rb ee/spec/lib/gitlab/duo_workflow/client_spec.rb
Edited by Angus Ryer

Merge request reports

Loading
Loading