🤖 *beep boop — clanker comment.* ## Summary The docker+machine fleet boots a custom COS image built by [linux-cos](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/linux-cos): `runners-cos-stable-v20230612-01-1` (amd64, COS 109, built 2024-05) and `runners-cos-arm64-stable-beta`. Every refresh is a project, #29264 being the current one. Of the seven Packer provisioners, one still does something (docker-cleanup, on the six MaxBuilds > 1 shards) and it can run from cloud-config. The rest are overwritten at boot, unreachable, stale, or fix a problem the build itself created. Decision: boot the ephemeral VMs from a copy of the pinned stock COS image with the existing cloud-config, and stop running Packer. amd64, arm64 and GPU compared below. Related: #29264, #29880. ## What the image adds | provisioner | status today | proposal | |---|---|---| | `10_apply_iptables_rules.sh` | VPC rules on the `docker-machine` tag already enforce ingress (22/2376 from managers only) and egress (deny to 10.0.0.0/8, blocklist). The rate limit chains are log-only and nothing reads the log. Metadata block: see below. | drop | | `11_setup_docker.sh` | cloud-config writes its own `daemon.json` to the same path at every boot. | drop | | `20_install_node_exporter.sh` | Nothing scrapes it, and no VPC rule allows 9100 to the VMs. | drop | | `20_install_docker_cleanup.sh` | Only matters where a VM runs many builds: `p-com`, `p-m`, `p-s` (40), `org-m`, `org-s` (10), `p-l` (5). | move to cloud-config as a systemd unit behind a `maxBuilds > 1` conditional, tag pinned | | `30_install_gpu_driver.sh` | The GPU shard's cloud-config already runs `cos-extensions install gpu` at boot. The baked driver is a cache hit for that step: 9s on the fleet image against 60s on stock (download of the current driver). The one pre-warming step that saves time, on one shard. | drop. The GPU shard takes about 50s more per VM on stock, absorbed by `idleCount: 7`. | | `90_pull_container_images.sh` | Layers from May 2024. With `pull_policy = always` every tag is resolved per job, and on amd64 all of them have moved: pulls take as long as on stock. On the three week old arm64 image `ruby:3.1` is still a hit. `ruby:3.1` is the runner default image and runs 154k jobs a day on `l-s-amd64` (6.3%), under 1% elsewhere. | drop from the image. `docker-prepull` unit in cloud-config for `ruby:3.1` on `l-s-amd64` only, pulling after docker-machine's restart so create time is unchanged (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29891#note_3900618661). | | `99_cleanup_account_state.sh` | Removes home directories the build created. | drop | ### Metadata server The image's `DOCKER-USER` rule blocks job containers from `169.254.169.254`. VPC firewall cannot do that, and it is not needed: the VM SA `ephemeral-runner@<shard-project>` has no IAM bindings at project, folder or org level, instance metadata is the public cloud-config, and every job container is `privileged` anyway. Nothing on the VM uses the SA, so the VMs can be created without one and the metadata server then has no token to serve. https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/194 adds `--google-no-service-account` for that. The GPU shard adds `devstorage.read_only` for `cos-extensions install gpu` and needs checking before losing its SA. ## Result, amd64 Runner sandbox, `t2d-standard-2`, fleet cloud-config on both images, three repeats per job (full numbers in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29891#note_3899787548). Stock `cos-125-lts` creates VMs faster (median 33s vs 36s), pulls every image faster, had no create failures against one on the fleet image, and runs the rspec and workhorse job bodies in the same time. DinD, MTU 1360, IPv6 on docker0 and swap all work from the cloud-config alone. ## Result, arm64 and GPU arm64 on `t2a-standard-2` (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29891#note_3900302238): same as amd64, stock creates faster (median 38s vs 41s) and pulls the same. One exception: `ruby:3.1`, the runner default image, is a cache hit on the fleet arm64 image (0.3s vs 11.8s) because that image is three weeks old and the tag is EOL. The pre-pull pays for tags that do not move while the image is fresh, which the copy pipeline could keep if it turns out to matter. GPU: passes on both images once docker-machine waits for cloud-init before configuring Docker, which the fleet GPU shard already does and the sandbox experiments did not. The flag as shipped fails on stock COS (`cloud-init status --wait` never exits 0 there, for COS image reasons). https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/195 waits on `cloud-init.target` instead and renames the flag to `--google-cos-wait-for-cloud-init`. `Machine created` with the wait: 1m18 to 1m37 on the fleet image, 1m49 to 2m13 on stock, the difference being the 60s driver download the baked driver avoids. Details in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29891#note_3900685843. ## Changes 1. linux-cos: replace the Packer build with the image copy below, one pin per arch. 2. argocd/apps and chef: `machineImage` to the copied image per shard, docker-cleanup unit in cloud-config, `docker-prepull` unit for `l-s-amd64`, remove the `05-iptables-restore-wants.conf` drop-in, `google-no-service-account` (and drop `google-service-account` and `google-scopes`). 3. Roll out shard by shard, small shards first. GPU shard last: switch its cloud-config to `--google-cos-wait-for-cloud-init`, quote `permissions`, drop `Requires=docker.service` from `gpu-driver.service`. 4. After the switch: `google-cos-tls-via-metadata` in `MachineOptions` and the TLS unit in the cloud-config, once docker-machine !196 is released and the manager image has a runner build with gitlab-runner !7476. About 6s off every create (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29891#note_3915801252). GPU shard: the TLS unit gets `After=gpu-driver.service`. ## Image pipeline after the switch Not the family pointer: it moves when Google publishes, so a bad COS release reaches new VMs without a deploy. Not a raw `cos-cloud` image name: Google marks it DEPRECATED at the next release and OBSOLETE later, after which VM creation fails, so an old pin cannot be rolled back to. Instead, linux-cos copies the pinned image into our project: ``` gcloud compute images create runners-cos-amd64-125-19216-655-40 \ --source-image=cos-125-19216-655-40 --source-image-project=cos-cloud \ --family=runners-cos-stable --project=gitlab-ci-155816 ``` Same disk content as Google's image, usable until we delete it, a few GB of storage per version. Not Packer: the googlecompute builder boots the source and captures the disk, which produces first-boot state like the UID collision in #29264. The name includes the COS version, which settles the provenance question there. Bumps are two MRs: linux-cos bumps `google_cos_version` per arch (scheduled job or Renovate custom datasource, there is no built-in one for GCE images), then argocd/apps and chef bump the image name per shard. Shards pin the name, not our family, so a copy is never a fleet-wide deploy.