Keep development-only settings out of default simulations
A default just up now reproduces a customer's install. Development-only
settings move behind one switch, just up --lane development, recorded in
instance state, so a lab result departs from the field only when the run asks
it to. ADR-0003 (docs/adr/0003-a-run-reproduces-a-customer-install.md)
records the decision and the alternatives considered.
Closes #9 (closed)
flowchart LR
U["just up"] --> L{"which lane?"}
L -->|"no lane flag: the default"| C["customer install<br/>preset env dropped<br/>gateway auth on<br/>mock model"]
L -->|"development"| D["development lane<br/>three gateway auth bypasses<br/>recorded in state"]
L -->|"any other value"| X["refused before<br/>anything is created"]
C --> S["aigw start reads the lane;<br/>SIM_AIGW_ENV off the lane's value<br/>warns and is recorded"]
D --> S- The AI gateway runs the auth setup from the GitLab Duo Self-Hosted offline
deployment doc: four self-signed JWT keys minted once per VM, workflow
service auth on, and both portal URLs pointed at the instance. The keys reach
podman through its environment, never its command line. The development lane
swaps in the three bypasses (
AIGW_AUTH__BYPASS_EXTERNAL,AIGW_AUTH__BYPASS_JWT_SIGNATURE,DUO_WORKFLOW_AUTH__ENABLED=false), and a flow runs on either lane. - caproni drops the caproni-demo preset's
global.extraEnvand its OpenBao debug logging, logs each dropped key, and adds back only what the simulator names. A values shape the parser does not positively understand fails the install and is named with its line. - A staging-minted license takes
SIM_LICENSE_MODE=stagingon every base. caproni renders the pair into the chart values at install, and caproni, omnibus and Geo recordlicense_modein state, at install or on a later license call. A failed activation without staging mode names it as the likely cause and prints thejust licenseline with the instance's add-on. SIM_AIGW_ENVstill changes any one auth setting on either lane.aigw startwarns when it moves one off the lane's value, records it asaigw_auth_override, and its start line reports the auth the container runs.- The mock model stays the default stand-in for the customer's model.
aigw-jwt-setupbecomes a deprecated no-op,up --edition cestarts no gateway, andaigw startrefuses a CE instance.
What the lab showed
Two instances on GitLab 19.4.1-ee with the mock model, both torn down
afterwards: caproni installed on the default lane with a staging license, and
omnibus installed with --lane development and no staging mode, then licensed
and moved between lanes. All 14 checks below passed. The one failure,
flows on a stock omnibus runner, predates this change (Known limits).
The 14 checks
| behaviour | result |
|---|---|
default caproni up with a staging license |
Ultimate, a self_hosted_dap seat on root, Agent Platform allowed; the license pair rendered at install, every Rails deployment still on revision 1 |
| preset environment | five keys and the OpenBao log level dropped and logged; only the staging pair left in the chart's global.extraEnv; AI_GATEWAY_URL, DEVELOPMENT_AI_GATEWAY_URL and CLOUD_CONNECTOR_SELF_SIGN_TOKENS absent from the webservice, Sidekiq and toolbox pods; cloud_connector_url is https://cloud.gitlab.com/ai |
| customer auth with a valid, a missing and a foreign-signed token | gateway 422, 401, 401; workflow service token call OK, UNAUTHENTICATED, UNAUTHENTICATED; the same on both bases |
SIM_AIGW_ENV bypass on the default lane |
the warning names the lane and each variable, state records it, and a request with no token is served; a single bypass reads partly bypassed and opens the gateway alone; a plain restart clears the record and restores 401 |
up --lane development on omnibus |
lane recorded, start line auth bypassed (--lane development), requests with no token served by both services |
SIM_AIGW_ENV on the development lane |
setting DUO_WORKFLOW_AUTH__ENABLED=true warns, is recorded, and closes the workflow service alone; the lane's own value does not warn |
| failed activation without staging mode | names staging mode and prints SIM_LICENSE_MODE=staging just license <code-or-path> self_hosted_dap; with staging mode set, no hint |
| a later license call with staging mode | omnibus reconfigures, licenses, assigns the seat and records license_mode=staging; caproni records it on an instance whose state lacked it |
--lane bogus and a bare --lane |
refused by name, with no VM and no state created |
| Cloud Connector key rotated while the gateway runs, after it has served requests | tokens signed by the new key refused by both services at once and four minutes later, accepted after aigw start |
| gateway JWT keys | files 600 root in a 700 directory; a missing key minted and moved in whole; a key cut short fails the start and leaves the running gateway alone; no key in the process list or the journal; real-model status prints none |
| CE | aigw start refused by name |
| flow execution on caproni | simulation/test finished with 5 checkpoints after install, after the bypass was restored, and after the key rotation |
just doctor |
no drift on either base |
Known limits
- On a stock omnibus instance a flow fails before its first checkpoint: the
shell executor's job cannot write the duo CLI to
/usr/local/bin. This predates the change and gets its own MR. - The workflow service fetches the instance keys at start, while the
gateway's HTTP side fetches them on its first authenticated request. A key
created between
aigw startand that request was accepted by the gateway and refused by the workflow service, soaigw startafter a rotation is still the step that brings both into line. - Not exercised live:
up --edition ce(the engine self-test covers it), Geo'slicense_moderecord, a failedkubectl set envor reconfigure on a later license call, and the 24-hour bound itself, which is the Cloud Connector library's default.
Testing
just ci passes, the engine self-test included. Lab evidence is above.
Claude Code assisted with the change, its tests and this description, and ran
the lab checks above. Each commit carries the AI-Assisted: yes and
AI-Tools: Claude Code trailers that CONTRIBUTING.md asks for.