Google COS: wait for cloud-init and fail closed on missing Docker network rules

Kam Kyralarequested to merge
kkyrala/google-cos-cloud-init-network-readiness into main

Summary

Add an opt-in readiness gate for Google Container-Optimized OS workers. When the GCE instance metadata key gitlab-docker-network-readiness-gate=true is set, docker-machine now:

  1. waits for cloud-init to finish before it provisions Docker;
  2. completes the normal Docker TLS configuration and final restart;
  3. verifies the expected Docker bridge NAT/FORWARD rules;
  4. restarts Docker once if the rules are missing;
  5. fails machine creation and stops Docker if the rules are still absent.

Runner only adds a machine to the available pool after docker-machine create returns successfully, so a worker that fails this gate cannot receive a job.

Why

GPU hosted-runner workers can enter a state where the host and Docker daemon are healthy enough to pull an image, but every network request from the job container fails. The first fix added After=iptables-restore.service to Docker, but a post-convergence canary worker still reproduced the total container-egress blackout.

Production timestamps exposed a second race on that worker:

  • VM creation began at 15:14:27.
  • docker-machine reported Docker is up and running / Machine created at 15:15:38.
  • Runner began using it at 15:15:40.
  • The Google COS provisioner waits for SSH and sudo docker version, but not for cloud-init, whose final stage installs the GPU driver after Docker starts.

This change closes that readiness gap and validates the final Docker rule state rather than assuming startup ordering alone is sufficient.

Related:

Safety

  • Opt-in only; no behavior changes unless the metadata key is exactly true.
  • No external image pull or public endpoint is required by the readiness check.
  • Missing metadata is treated as disabled only on a confirmed HTTP 404; metadata transport failures fail provisioning rather than silently bypassing an explicitly enabled gate.
  • Cloud-init wait is bounded at five minutes.
  • Docker rules are polled before and after one repair restart.
  • Persistent failure stops Docker and returns an error from provisioning.

The existing Docker After=iptables-restore.service change remains useful defense in depth: it removes the known initial ordering race. This gate handles late boot work and protects the worker readiness boundary.

Validation

  • go test ./libmachine/provision ./drivers/google
  • Added tests for gate enabled/disabled, healthy rules, successful one-restart repair, and fail-closed persistent failure.
  • Fresh cross-model review found no remaining blocker/should-fix findings after fixes for metadata error handling, exact rule checks, bounded polling, fail-closed cleanup, and bounded cloud-init wait.

Rollout

Canary via the existing GPU slot-1 roles on both colors, then verify:

  • no job is assigned before cloud-init/readiness completes;
  • healthy worker creation remains within the provisioning budget;
  • injected missing Docker rules trigger exactly one restart and recovery;
  • persistent injected failure prevents the VM from entering Runner's available pool;
  • corrected clone/pcap blackout rate falls to zero on gated slots.

Kill criteria

Disable the metadata flag immediately if any of these occur:

  • healthy workers are rejected by the rule check;
  • provisioning latency or queueing regresses materially;
  • a job is assigned before readiness passes;
  • the repair restart interrupts an assigned job;
  • Docker or GPU availability regresses.

Merge request reports