Add server URL and cache network probes to the boot-verify canary

Igorrequested to merge
boot-verify-network-probes into main

What does this MR do?

Adds two opt-in network probes to the experimental boot-verify canary. Both run in a dedicated boot_verify_probes build stage of the synthetic job, executed only for synthetic builds, in the helper environment on the created machine. That is the network location real checkouts and cache transfers use, which matters because manager and worker network paths differ. Both default to off, so existing configurations keep today's behavior.

[runners.experimental.boot_verify]
  enabled = true
  verify_server_url = true          # new, default false
  verify_cache = true               # new, default false
  # new, default empty
  verify_commands = ["LD_LIBRARY_PATH=/usr/local/nvidia/lib64 /usr/local/nvidia/bin/nvidia-smi"]

verify_server_url

Probes the Git HTTP endpoint under the effective clone_url (falling back to url): a GET of info/refs on a nonexistent repository, which GitLab answers with 401 before any existence check. Any HTTP response passes. DNS, routing, timeout, and TLS failures fail the canary, so the manager never reports Ready. TLS is verified the way a real checkout verifies it: against the runner's tls-ca-file chain when configured, otherwise the system roots. verify_server_url_insecure opts out. A new probe-url helper subcommand makes the request, because curl is not in the helper images and git or wget exit codes cannot tell "server said 404" from "no route to server".

verify_cache

Writes a canary object with the runner's real cache credentials and reads it back. The key is unique per boot, so the read only succeeds if this boot's write persisted, and concurrent manager replicas cannot interfere. Objects are a few hundred bytes and are never deleted. Retention comes from the bucket's lifecycle policy, and the production buckets already have one (14 days on gitlab-com-runners-cache, 30 on the distribution buckets).

verify_commands

Shard-owned commands appended to the canary job's script, run in the job container with the runner's configured devices and volumes, exactly as customer job containers are built. Motivated by a GPU shard that shipped with accelerators attached but no driver on the VM image: the canary passed and every customer job failed with "NVIDIA Driver was not detected". There a single nvidia-smi invocation (the exact form validated in the sandbox, see the config example) fails on a missing host driver, missing device nodes, or missing library mounts. The commands run on the runner's default image and must be runnable there: nvidia-smi lives on the library mount outside PATH, needs its matching libnvidia-ml on the loader path, and is a glibc binary. The list is explicit configuration rather than inferred from machine options, which would couple the canary to driver-specific knowledge. The commands share the script step's timeout: per-command timeout(1) wrappers would fail on job images that lack the binary, and a hung command still fails the canary at the step deadline.

The network probes run through the helper binary and need a helper image matching the runner version. A passing canary now also dumps its build output at debug log level.

Why was this MR needed?

The canary runs its synthetic job with GIT_STRATEGY: none and a bare echo script, so it never touches the network paths or host resources a real job needs. During a recent incident, two runner shards had worker VPCs with no route to the ci-gateway their clone_url pointed at. The canary passed, the managers reported Ready, and every real job failed at checkout for twelve hours (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29588, confidential). A cache bucket the workers cannot access, or a GPU shard whose VMs boot without a working driver, are the same class of gap.

What's the best way to test this MR?

Unit tests cover the probe command and the script generation. I validated it end to end against real GCP infrastructure: a 3-replica docker+machine deployment on GKE with both probes on, an unroutable clone_url as the negative case, and a regular job afterwards. Results with trace excerpts are in the discussion below.

What are the relevant issue numbers?

Edited by Igor

Merge request reports