Run Duo Agent Platform E2E specs on a Caproni CNG deploy

Igor Drozdovrequested to merge
635744-dap-e2e-on-cng into master

What does this MR do?

Adds cng-duo-agent-platform to the CNG E2E child pipeline. The job runs the Duo Agent Platform flow specs against a Caproni-deployed CNG instance.

The omnibus duo-agent-platform job is the only E2E coverage that runs a Duo flow to completion. It is manual in most merge request pipelines, so checkpoint changes could merge without it. The omnibus job stays as it is.

The job deploys the chart's ai-gateway subchart with the agentic-mock Duo Workflow Service, activates a cloud license with QA_EE_ACTIVATION_CODE, and runs QA::EE::Scenario::Test::Integration::DuoAgentPlatform. Rails and Workhorse reach both services through the gitlab-ai-gateway Service in the cluster. The flow's CI job runs outside the cluster, but duo-cli connects to Workhorse, not to the Duo Workflow Service. On pipeline::tier-3, the job also runs when an MR changes Workhorse's Duo Workflow proxy. Unlike the other Caproni jobs, it runs the QA runner's job containers on the DinD bridge network; see below. qa/caproni/README.md covers the details.

The job is non-blocking for now. It inherits allow_failure: true from .caproni-test, so a failure shows as a warning and the CNG child pipeline still passes. Once it has a few passing runs, set allow_failure: false on the job, as cng-instance-ff-inverse and cng-vue3-rollout do. A failure then fails the child pipeline, the e2e:test-on-cng trigger job (strategy: depend), and the MR pipeline. The AI Gateway image is latest, built outside this repository, so once the job blocks, a Duo Workflow Service regression on its main would fail matching MRs here until it's fixed there. The README notes this trade-off for whoever flips it.

The job passed in https://gitlab.com/gitlab-org/gitlab/-/jobs/17424754847: both specs in 2m26s, run on their own through a temporary commit that has since been dropped.

When a flow pipeline doesn't succeed, the specs now log the workload jobs' traces before teardown deletes the project. That trace is how the sandbox problem below was found.

Why the job uses the bridge network

On the host network, every flow job failed with exit code 65 before it reached Workhorse:

WebSocket upgrade rejection body: Connection to gitlab.172.18.0.2.nip.io blocked: resolved to one of this host's addresses
Cause and options

The 403 comes from the egress proxy of the flow job's sandbox, SRT (sandbox-runtime), not from GitLab. SRT refuses connections from allowlisted hostnames that resolve to an address on one of its host's own network interfaces. In this rig, GitLab's address is the DinD service's own IP (getent hosts docker), where k3d publishes port 80. With QA_DOCKER_NETWORK: host, the flow job shares DinD's network, so GitLab's address is the job's own. On omnibus, GitLab runs in its own container with its own IP, so the check never triggers.

Options:

  1. Run the job containers on the DinD bridge network (chosen). QA_DOCKER_NETWORK: bridge on this job only. The job container gets its own IP and reaches GitLab through the port k3d publishes on DinD. One line, no product workaround.
  2. Allowlist the IP literal. SRT exempts an address listed in its allowlist as an IP literal. But AiDomainSettingsInstanceUpdate rejects IP literals (VALID_DOMAIN_REGEX requires a letter TLD), so the seeder would have to write Ai::Setting#allowed_domains through the Rails console and skip that validation.
  3. Give GitLab an address that isn't DinD's. The QA process outside DinD can only reach DinD's IP, so this means reworking the rig's networking.

This may also affect real self-managed setups: a runner that shares the GitLab host's network, such as a Docker runner with host networking on the GitLab server, hits the same block. Admins can't work around it in the domain settings, because they don't accept IP literals. Tracked in #636091.

Related issues

Checklist

Pre-merge

Consider the effect of the changes in this merge request on the following:

If new jobs are added:

  • Change-related rules (e.g. frontend/backend/database file changes): changes to the scenario, its specs, its Caproni fragment and seeder, or the shared scripts and CI template that wire them in (seed.rb, write_local_config.rb, caproni.gitlab-ci.yml) at any tier. Changes to Duo Agent Platform code under ee/app/**/ai/duo_workflows/, ee/lib/ai/duo_workflow{,s}/, ee/lib/api/ai/duo_workflows/, ee/lib/gitlab/duo_workflow/, and Workhorse's Duo Workflow proxy under workhorse/internal/ai_assist/duoworkflow/ on pipeline::tier-3. Also the pipeline:run-all-e2e label.
  • Frequency they are running (MRs, main branch, nightly, bi-hourly): matching MRs, plus scheduled master pipelines other than the bi-hourly maintenance schedule
  • Add a duration chart to https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations if there are new jobs added to merge request pipelines

This will help keep track of expected pipeline time and cost increases.

Post-merge

Edited by Igor Drozdov

Merge request reports