Fix the simulator causes of two self-hosted verifier failures
verify-self-hosted now passes its model check on a mock caproni instance and says which side refuses its WebSocket check. On a stock instance it reported two failures. The model check failed because of how the simulator registered the mock model. The WebSocket check failed first because pods could not resolve the instance's own name, and then because the verifier's own request is refused, which this MR reports rather than fixes.
Closes #12 (closed)
flowchart LR
V["verify-self-hosted<br/>in the toolbox pod"] --> M["model check<br/>GET endpoint/v1/models"]
V --> W["WebSocket check<br/>GET external URL"]
M --> M1["before: the gateway's loopback,<br/>which is the pod: refused"]
M --> M2["after: model list on the node: 200"]
W --> D1["before: name not resolvable in pods"]
W --> D2["after: CoreDNS maps it to the node"]
D2 --> O["Workhorse origin check"]
O --> R["verifier request<br/>Host with :80, Origin without: 403"]
O --> S["standard request<br/>Host and Origin match: 101"]- caproni pods resolve the instance's own URL.
upadds a CoreDNS server block that maps the GitLab host to the node, where the ingress listens. k3s CoreDNS already uses thehostsplugin in its main block and refuses a second one in the same block, so the entry is a*.serverkey; an*.overridekey crash-loops CoreDNS. - The mock model's endpoint is a static model list served from the VM, on the address GitLab reaches the gateway at. The mock answers inside the gateway and never calls its endpoint, so the list exists for the verifier, which probes a customer's model server at
/v1/models. It runs as a transientsim-model-listunit on port 5053 (SIM_MODEL_LIST_PORT).self-hosted-modelwith a URL is unchanged. verify-self-hostedreplays the verifier's handshake, read off its check class, beside a standard one, and prints which side refuses. The verifier's request is gitlab-org/gitlab#628161 (closed); once its fix lands, both handshakes upgrade and the output says so.gce.mdand the CHANGELOG describe the three changes.
What the lab showed
One caproni instance on GitLab 19.4.1-ee with a staging license, the mock model and the default lane, torn down afterwards.
The checks
| behaviour | main |
this branch |
|---|---|---|
| model check | ECONNREFUSED to the gateway's loopback |
endpoint accessible |
| WebSocket check | name resolution failed for the instance's own URL | 403 |
| WebSocket output after the verifier | none | standard request 101, verifier request 403, and a verdict naming the Host and Origin mismatch |
developer/v1 with the new endpoint |
not run | finished; the model list received five requests, all GET /v1/models, none while the flow ran |
With only the DNS entry in place on main, Workhorse logged request origin not allowed by Upgrader.CheckOrigin for the verifier's request while Rails authorized it. A Host and Origin pair that agree on the port upgraded whether or not both carried :80, and a pair that disagreed was refused either way.
Known limits
- The CoreDNS entry and the model list ran through their own functions on a running instance, not through a fresh
up. The engine self-test pins thatupapplies the entry on caproni only. - On omnibus the model list binds
127.0.0.1; it has not run there. - The LiteLLM proxy arm still registers a loopback endpoint, which a caproni pod cannot reach. That arm is unchanged here.
Testing
just ci passes. The engine self-test covers the CoreDNS block, the model list and the WebSocket split against a stubbed endpoint, and fails against main.
Claude Code wrote this change, its tests and this description, and ran the checks above in a simulated environment. Each commit carries the AI-Assisted: yes and AI-Tools: Claude Code trailers that CONTRIBUTING.md asks for.