Keep development-only settings out of default simulations

A default just up now reproduces a customer's install. Development-only settings move behind one switch, just up --lane development, recorded in instance state, so a lab result departs from the field only when the run asks it to. ADR-0003 (docs/adr/0003-a-run-reproduces-a-customer-install.md) records the decision and the alternatives considered.

Closes #9 (closed)

flowchart LR
    U["just up"] --> L{"which lane?"}
    L -->|"no lane flag: the default"| C["customer install<br/>preset env dropped<br/>gateway auth on<br/>mock model"]
    L -->|"development"| D["development lane<br/>three gateway auth bypasses<br/>recorded in state"]
    L -->|"any other value"| X["refused before<br/>anything is created"]
    C --> S["aigw start reads the lane;<br/>SIM_AIGW_ENV off the lane's value<br/>warns and is recorded"]
    D --> S
  • The AI gateway runs the auth setup from the GitLab Duo Self-Hosted offline deployment doc: four self-signed JWT keys minted once per VM, workflow service auth on, and both portal URLs pointed at the instance. The keys reach podman through its environment, never its command line. The development lane swaps in the three bypasses (AIGW_AUTH__BYPASS_EXTERNAL, AIGW_AUTH__BYPASS_JWT_SIGNATURE, DUO_WORKFLOW_AUTH__ENABLED=false), and a flow runs on either lane.
  • caproni drops the caproni-demo preset's global.extraEnv and its OpenBao debug logging, logs each dropped key, and adds back only what the simulator names. A values shape the parser does not positively understand fails the install and is named with its line.
  • A staging-minted license takes SIM_LICENSE_MODE=staging on every base. caproni renders the pair into the chart values at install, and caproni, omnibus and Geo record license_mode in state, at install or on a later license call. A failed activation without staging mode names it as the likely cause and prints the just license line with the instance's add-on.
  • SIM_AIGW_ENV still changes any one auth setting on either lane. aigw start warns when it moves one off the lane's value, records it as aigw_auth_override, and its start line reports the auth the container runs.
  • The mock model stays the default stand-in for the customer's model. aigw-jwt-setup becomes a deprecated no-op, up --edition ce starts no gateway, and aigw start refuses a CE instance.

What the lab showed

Two instances on GitLab 19.4.1-ee with the mock model, both torn down afterwards: caproni installed on the default lane with a staging license, and omnibus installed with --lane development and no staging mode, then licensed and moved between lanes. All 14 checks below passed. The one failure, flows on a stock omnibus runner, predates this change (Known limits).

The 14 checks
behaviour result
default caproni up with a staging license Ultimate, a self_hosted_dap seat on root, Agent Platform allowed; the license pair rendered at install, every Rails deployment still on revision 1
preset environment five keys and the OpenBao log level dropped and logged; only the staging pair left in the chart's global.extraEnv; AI_GATEWAY_URL, DEVELOPMENT_AI_GATEWAY_URL and CLOUD_CONNECTOR_SELF_SIGN_TOKENS absent from the webservice, Sidekiq and toolbox pods; cloud_connector_url is https://cloud.gitlab.com/ai
customer auth with a valid, a missing and a foreign-signed token gateway 422, 401, 401; workflow service token call OK, UNAUTHENTICATED, UNAUTHENTICATED; the same on both bases
SIM_AIGW_ENV bypass on the default lane the warning names the lane and each variable, state records it, and a request with no token is served; a single bypass reads partly bypassed and opens the gateway alone; a plain restart clears the record and restores 401
up --lane development on omnibus lane recorded, start line auth bypassed (--lane development), requests with no token served by both services
SIM_AIGW_ENV on the development lane setting DUO_WORKFLOW_AUTH__ENABLED=true warns, is recorded, and closes the workflow service alone; the lane's own value does not warn
failed activation without staging mode names staging mode and prints SIM_LICENSE_MODE=staging just license <code-or-path> self_hosted_dap; with staging mode set, no hint
a later license call with staging mode omnibus reconfigures, licenses, assigns the seat and records license_mode=staging; caproni records it on an instance whose state lacked it
--lane bogus and a bare --lane refused by name, with no VM and no state created
Cloud Connector key rotated while the gateway runs, after it has served requests tokens signed by the new key refused by both services at once and four minutes later, accepted after aigw start
gateway JWT keys files 600 root in a 700 directory; a missing key minted and moved in whole; a key cut short fails the start and leaves the running gateway alone; no key in the process list or the journal; real-model status prints none
CE aigw start refused by name
flow execution on caproni simulation/test finished with 5 checkpoints after install, after the bypass was restored, and after the key rotation
just doctor no drift on either base

Known limits

  • On a stock omnibus instance a flow fails before its first checkpoint: the shell executor's job cannot write the duo CLI to /usr/local/bin. This predates the change and gets its own MR.
  • The workflow service fetches the instance keys at start, while the gateway's HTTP side fetches them on its first authenticated request. A key created between aigw start and that request was accepted by the gateway and refused by the workflow service, so aigw start after a rotation is still the step that brings both into line.
  • Not exercised live: up --edition ce (the engine self-test covers it), Geo's license_mode record, a failed kubectl set env or reconfigure on a later license call, and the 24-hour bound itself, which is the Cloud Connector library's default.

Testing

just ci passes, the engine self-test included. Lab evidence is above.

Claude Code assisted with the change, its tests and this description, and ran the lab checks above. Each commit carries the AI-Assisted: yes and AI-Tools: Claude Code trailers that CONTRIBUTING.md asks for.

Merge request reports

Loading
Loading