Fix platform key breaking image inspect on Podman backend

Lachlan Grantrequested to merge
39608-docker-platform-podman-api-version into main

What does this MR do?

Since the moby/moby v29 SDK migration, ImageInspectWithRaw is called with a platform argument to disambiguate multi-arch images in the local cache. The SDK hard-errors client-side when a platform is requested against a daemon whose negotiated API version is below 1.49. Podman's Docker-compat layer tops out at API 1.41, so any job setting a docker image platform against a Podman backend failed on every pull:

"platform" requires API version 1.49, but the Docker daemon API version is 1.41

The pull manager now falls back to a platform-agnostic inspect (matching pre-migration behavior) when the daemon doesn't support the platform parameter, instead of letting the SDK's gate fail the job outright. ImagePull and ContainerCreate are unaffected, since the SDK does not gate those calls.

Update: originally ImageInspectWithRaw forced its own API version negotiation to make this decision. Per @ajwalker's review below, that duplicated work the docker executor already does at connect time (see connectDocker in executors/docker/docker.go). The decision now lives in the pull manager (inspectOptions in executors/docker/internal/pull/manager.go), using the executor's already-known serverAPIVersion -- no extra negotiation round trip. ImageInspectWithRaw itself is back to a thin pass-through matching the official SDK's shape (platform via ImageInspectOption, not a required parameter).

While verifying this fix against real Podman, a second, unrelated, pre-existing bug turned up in the same file (parsePlatform resolving bare architecture strings to the runner host's own OS instead of always linux). Per GitLab's minimum-viable-change policy, that's tracked and fixed separately, not in this MR: #39622 (closed) / !6993 (merged)

Closes #39608 (closed)

Testing

  • Unit tests cover both sides of the version gate directly (TestInspectOptions* in executors/docker/internal/pull/manager_test.go), plus the ImageInspectWithRaw pass-through itself (helpers/docker/official_docker_client_test.go).
  • Added TestNewPullManagerConfigWiresServerAPIVersion (executors/docker/pull_test.go) covering the one link in this chain not already covered elsewhere: that connectDocker's negotiated version actually reaches the pull manager's config.
  • TestPodmanCommandWithPlatformKey (executors/docker/docker_podman_integration_test.go, from the split-out !6991 (merged)) is an existing opt-in end-to-end regression test for this exact bug, through the real executor against a real Podman daemon. Passes on this branch.
  • go build/vet (both normal and -tags integration), the full helpers/docker/executors/docker test trees, and make lint all pass.
  • Reproduced and verified against a real, local Podman daemon (not a mock) using the script below, and again via an interactive walkthrough in the pinned comment below.

Reproducing against real Podman

This performs the exact pull-then-inspect sequence executors/docker/internal/pull/manager.go runs for a job with image:docker:platform: set, using GitLab Runner's own helpers/docker client, against a real Podman socket. It runs the sequence once against origin/main (unpatched, in a disposable git worktree) to reproduce the failure, then again on this branch to show it's fixed.

Requires podman (with podman machine on macOS) and go.

reproduce.sh

 
#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
FIXED_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"

step() {
	echo
	echo "=================================================================="
	echo "STEP $1: $2"
	echo "=================================================================="
}

step 1 "Locating a running Podman machine"
if [ "$(podman machine list --format '{{.Running}}' --noheading 2>/dev/null | head -1)" = "true" ]; then
	echo "A Podman machine is already running -- reusing it."
else
	echo "No Podman machine is running. Starting the default one now"
	echo "(this boots a small Linux VM and can take up to a minute)..."
	podman machine start
fi

SOCK="$(podman machine inspect --format '{{.ConnectionInfo.PodmanSocket.Path}}')"
echo
echo "Podman's Docker-compatible API socket is at:"
echo "  $SOCK"

step 2 "Asking the real Podman daemon what Docker API version it speaks"
echo "Running: curl --unix-socket \$SOCK http://d/version"
API_VERSION="$(curl -s --unix-socket "$SOCK" http://d/version | python3 -c 'import json,sys; print(json.load(sys.stdin)["ApiVersion"])')"
echo
echo "Daemon reports API version: $API_VERSION"
echo "The SDK's platform-filtered inspect requires API version 1.49 or later."
echo "$API_VERSION < 1.49, so this daemon should trip the bug below."

run_repro() {
	local dir="$1"
	(cd "$dir" && go run ./tmp_repro39608 "unix://$SOCK")
}

step 3 "Reproducing the bug: running the pull+inspect sequence against UNPATCHED code (origin/main)"
echo "Checking out origin/main into a disposable git worktree so your"
echo "current branch/working tree is left untouched..."
PREFIX_WORKTREE="$(mktemp -d)/gitlab-runner-prefix"
git -C "$REPO_ROOT" worktree add "$PREFIX_WORKTREE" origin/main --detach --quiet
cp -r "$SCRIPT_DIR" "$PREFIX_WORKTREE/tmp_repro39608"
echo "Worktree ready at: $PREFIX_WORKTREE"
echo
echo "Running the reproduction program against unpatched code..."
if run_repro "$PREFIX_WORKTREE"; then
	echo
	echo "!! UNEXPECTED: unpatched code succeeded. The bug may already be"
	echo "!! fixed upstream, or this Podman daemon reports API >= 1.49."
	UNPATCHED_RESULT="succeeded (unexpected)"
else
	echo
	echo "Confirmed: unpatched code fails with the exact error reported in"
	echo "https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39608"
	UNPATCHED_RESULT="failed as expected"
fi
echo
echo "Cleaning up the disposable worktree..."
git -C "$REPO_ROOT" worktree remove "$PREFIX_WORKTREE" --force

step 4 "Verifying the fix: running the same sequence against branch '$FIXED_BRANCH'"
if run_repro "$REPO_ROOT"; then
	FIXED_RESULT="succeeded as expected"
else
	FIXED_RESULT="!! FAILED (unexpected -- fix did not work)"
fi

step 5 "Summary"
echo "Podman daemon API version: $API_VERSION"
echo "origin/main (unpatched):   $UNPATCHED_RESULT"
echo "$FIXED_BRANCH (fixed): $FIXED_RESULT"

main.go (companion Go program, saved alongside reproduce.sh)

 
// Scratch reproduction program for


// https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39608


//


// It performs exactly the sequence executors/docker/internal/pull/manager.go


// runs for a job with `image:docker:platform:` set:


//


//  1. ImagePullBlocking(imageName)              -- download the image


//  2. ImageInspectWithRaw(imageName, platform)  -- look it up again, filtered


//     by platform, to return the right InspectResponse to the caller.


//


// Step 2 is the exact call (executors/docker/internal/pull/manager.go:309)


// named in the ticket's error: `inspecting image ... after pull`. This


// program calls the same helpers/docker.Client the real executor uses, so


// running it against a real Docker/Podman socket exercises the identical


// code path a CI job would hit.


package main

import (
	"context"
	"fmt"
	"os"

	mobyclient "github.com/moby/moby/client"
	v1 "github.com/opencontainers/image-spec/specs-go/v1"

	"gitlab.com/gitlab-org/gitlab-runner/helpers/docker"
)

func main() {
	if len(os.Args) < 2 {
		fmt.Fprintln(os.Stderr, "usage: repro39608 <docker-host, e.g. unix:///path/to/podman.sock>")
		os.Exit(2)
	}
	host := os.Args[1]

	dc, err := docker.New(docker.Credentials{Host: host})
	if err != nil {
		fmt.Fprintln(os.Stderr, "failed to create client:", err)
		os.Exit(1)
	}

	ctx := context.Background()
	const image = "docker.io/library/alpine:latest"
	platform := &v1.Platform{Architecture: "arm64", OS: "linux"}

	fmt.Println("  -> ImagePullBlocking(\"" + image + "\") ...")
	if err := dc.ImagePullBlocking(ctx, image, mobyclient.ImagePullOptions{}); err != nil {
		fmt.Fprintln(os.Stderr, "  -> pull failed:", err)
		os.Exit(1)
	}
	fmt.Println("  -> pull OK")

	fmt.Println("  -> ImageInspectWithRaw(\"" + image + "\", platform) ... (this is the manager.go:309 call from the ticket)")
	res, _, err := dc.ImageInspectWithRaw(ctx, image, platform)
	if err != nil {
		fmt.Println("  -> FAILED:", err)
		os.Exit(1)
	}
	fmt.Println("  -> SUCCESS: inspected image ID", res.ID)
}
Example output (real run, Podman 5.8.2, reporting API 1.44) 
STEP 2: Asking the real Podman daemon what Docker API version it speaks
Daemon reports API version: 1.44
The SDK's platform-filtered inspect requires API version 1.49 or later.
1.44 < 1.49, so this daemon should trip the bug below.

STEP 3: Reproducing the bug: running the pull+inspect sequence against UNPATCHED code (origin/main)
  -> ImagePullBlocking("docker.io/library/alpine:latest") ...
  -> pull OK
  -> ImageInspectWithRaw("docker.io/library/alpine:latest", platform) ...
  -> FAILED: "platform" requires API version 1.49, but the Docker daemon API version is 1.44 (main.go:54:0s)
Confirmed: unpatched code fails with the exact error reported in the issue.

STEP 4: Verifying the fix: running the same sequence against branch '39608-docker-platform-podman-api-version'
  -> ImagePullBlocking("docker.io/library/alpine:latest") ...
  -> pull OK
  -> ImageInspectWithRaw("docker.io/library/alpine:latest", platform) ...
  -> SUCCESS: inspected image ID sha256:1991bd789d7184290c3cce84fd6af068b8b745e9bddf178661ce7f5ecf68135c

STEP 5: Summary
Podman daemon API version: 1.44
origin/main (unpatched):   failed as expected
39608-docker-platform-podman-api-version (fixed): succeeded as expected

Two other pieces of work were split into separate MRs to keep this one minimal (GitLab's minimum-viable-change policy):

  • A general-purpose Podman integration-test suite (not specific to this fix): !6991 (merged)
  • The unrelated, pre-existing parsePlatform host-OS bug found while verifying this fix: !6993 (merged)
Edited by Lachlan Grant

Merge request reports