google: deliver the Docker TLS material as instance metadata, no SSH in the create path

Summary

Adds --google-cos-tls-via-metadata. With the flag, the server certificate and the dockerd drop-in are generated before the VM exists and attached to the insert request as instance metadata, together with the CA and the SSH key. A unit in the VM's cloud-config installs them. docker-machine does not open SSH during the create. It waits for a TLS handshake on port 2376 and then runs the usual connection check.

Tracking: gitlab-com/gl-infra/production-engineering#29891 (comment 3900726062) (option 2).

Why

The server certificate has the VM's IP in it, and the IP only exists after the insert. So the create path today is: wait for SSH, stop dockerd, copy three PEM files and a drop-in over SSH, restart dockerd, poll for the port every 3 seconds. On a stock COS 125 VM in the sandbox that keeps dockerd down from 7 to 13 seconds after boot, and the create takes 31 seconds at the median, with a long tail from the randomised sleeps in the wait loops.

Issuing the certificate for the machine name removes the dependency on the IP. The name is known before the insert, so everything dockerd needs can go into the insert request. The client dials the IP as before and verifies the certificate against the name.

What changes for users

The server certificate has the machine name in its SANs on both paths now. regenerate-certs on a machine created with the flag produces a certificate with the name and the IP.

Machines created with the flag have ServerName in their config.json. ls, config, env and the create-time check verify against it. Machines created without the flag have it empty and verify against the IP as before. The runner reads it from inspect (gitlab-org/gitlab-runner!7476 (merged)). The docker CLI has no equivalent option, so docker-machine env on its own is no longer enough for it. That is in the docs.

The SSH key is attached at insert time on both paths, which removes a SetMetadata call and its operation wait from every create. --google-use-existing keeps using SetMetadata.

With the flag, start refuses to re-create a machine from its disk when the instance is gone, the way bulkInsert mode already does. The certificates only exist in the original insert's metadata and COS keeps /etc on a tmpfs overlay, so a new instance on the old disk would boot without them.

The COS readiness gate and readiness URL are not checked with the flag, since the provisioner does not run. The flag warns when they are set together. The cloud-config has to order the TLS unit after whatever the VM has to finish before it is ready.

What the VM has to do

On stock COS, dockerd starts before cloud-init runs. cos-critical.target is ordered after docker.service and before cloud-init-local.service, so nothing delivered through user-data can affect the first dockerd start. The unit fetches the four metadata attributes, writes the PEM files and the drop-in, adds the iptables rule for 2376, and restarts dockerd. That happens about 8.5 seconds after boot and dockerd is back with the TLS listener half a second later, before anything else on the VM uses docker. The sandbox cloud-config is in gitlab-runner-sandbox, experiment cos-metadata-tls. docs/drivers/gce.md lists the metadata keys and what the unit has to do with them.

The server key is readable from inside the VM through the metadata server. That is the same exposure /etc/docker/server-key.pem has today for any privileged container.

Measurements

Sandbox, stock cos-125-lts, t2d-standard-2, bulkInsert in us-east1, the two experiments in the same 15 minutes with 20 fresh-VM jobs each plus idle and canary creates. Machine created durations in seconds:

	n	min	p50	mean	p90	max
--google-cos-tls-via-metadata	23	22.3	24.5	24.8	26.1	28.7
SSH provisioning (main)	23	27.9	30.8	31.7	36.0	36.7

Of the 24 seconds on the metadata path, about 5 are the bulkInsert call, 9 pass until the VM starts booting, and 9 more until the TLS listener is up. docker-machine connects within 150 ms of that. Boot-verify passed on both experiments and all 40 jobs succeeded.

Editing ServerName in a machine's config.json to another name makes docker-machine config fail with certificate is valid for <name>, localhost, not <other>. Clearing it fails with doesn't contain any IP SANs. ssh and regenerate-certs work on machines created with the flag.

Not in this MR

The fleet image bakes its own docker setup, which conflicts with the drop-in delivered here. The fleet rollout is a separate step after the stock COS switch.