test(embed): fail when the integration gate is on but the runtime is absent

Matt Cockaynerequested to merge
test/fail-on-contradictory-gate into main

A silent skip and a passing run look identical in a job summary. That is how the coverage gap spec 0012 recorded stayed invisible, and it is about to matter: the CI wiring for spec 0013 is imminent, and without this, setting INT_TEST_EMBED=1 against an image with no runtime would look exactly like success.

Two ways not to run, and only one is legitimate

Condition Before After
INT_TEST_EMBED unset skip skip — nobody asked; correct
Gate on, PHPBOTSCOUT_ORT_LIB empty skip fail — asked for, and cannot run

The second is a contradiction. You requested the integration tests and the dependency is absent; reporting that as a skip means a green pipeline for work that did not happen.

Why in the helper, not as a skip-counter in CI

The obvious alternative is counting --- SKIP lines in the job and failing if any appear. It is worse in three ways:

  • It cannot tell the two cases apart. A counter sees a skip, not a reason — so it fires on the ordinary, legitimate case (gate off, which is most runs and every developer's laptop) and gets disabled within a week. A check that gets disabled is worse than no check, because its presence implies coverage it is not providing.
  • It does not travel. Every repo gating a suite behind an env var would rebuild the same log-parsing in its own pipeline. In the helper it is one condition, in one place, and it moves with the code.
  • It is not available where it is most useful — a developer running go test locally with the gate on gets the same clear failure, rather than discovering it in CI.

Verified in all three states

gate off                    --- SKIP                       (legitimate)
gate on, no library         --- FAIL, rc=1                 (the bug, now caught)
gate on, library present    38 pass, 0 skip, 0 fail        (go-tools:v0.1.0)

The failure names the remedy, because an error a reader cannot act on is half an error:

PHPBOTSCOUT_ORT_LIB is not set, but the integration gate is on.

These tests were asked for and cannot run, which is a failure rather than
something to skip past: a silent skip here is a green pipeline for work that
did not happen.
Set PHPBOTSCOUT_ORT_LIB to a libonnxruntime path, or unset the gate to skip
honestly.

What this does not change

Nothing about the default developer experience. go test ./... with no gate set skips exactly as before — the ordinary case is untouched. Only a run that has asked for the integration tests can now fail on their absence.

Context

This is the half of spec 0013's remaining CI work that does not depend on the toolchain image. The other half — setting INT_TEST_EMBED=1 and PHPBOTSCOUT_ORT_LIB — waits on cicd!225 (merged), which moves the Go components onto go-tools:v0.1.0 (the image that carries the runtime; validated at 38/0/0 with the golden vector unmoved). Landing this first means that wiring MR is just two variables, with the guard already in place.

Merge request reports