Skip to content

FIPS runner, docker executor using Podman can't find local public docker image

Summary

The gitlab-runner is able to pull the image from docker hub but not from a local, public harbor registry (no password needed for pulling).

  • When using harbor1.mydomain.com/dh/library/swift:focal as image, I get the error described below.
  • When using swift:focal the runner is able to pull and execute the pipeline.
  • When using HARBORS_LOCAL_IP_ADDRESS/dh/library/swift:focal the runner throws the same "Failed to pull ..." error messages.
  • When ssh into the gitlab-runners VM, and executing podman pull --tls-verify=false harbor1.mydomain.com/dh/library/swift:focal, that works perfectly fine. After the image was manually pulled, I can re-run the pipeline, which now succeeds.

As a side note harbor1.mydomain.com is only defined in the internally used DNS server and is not accessible over the internet. Maybe the runner is ignoring the VM's /etc/resolf.conf or trying to access the registry over the internet.

Steps to reproduce

Working:

.gitlab-ci.yml 
default:
  tags:
    - podman
  image: swift:focal

Not working:

.gitlab-ci.yml 
default:
  tags:
    - podman
  image: harbor1.mydomain.com/dh/library/swift:focal

Actual behavior

The runner can't find and pull the image from a local, public harbor registry.

Expected behavior

The runner finds and pulls the image from a local, public harbor registry.

Relevant logs and/or screenshots

job log 
Preparing the "docker" executor
Using Docker executor with image harbor1.mydomain.com/dh/library/swift:focal ...
Pulling docker image harbor1.mydomain.com/dh/library/swift:focal ...
WARNING: Failed to pull image with policy "always": Error: No such image: harbor1.mydomain.com/dh/library/swift:focal (manager.go:239:0s)
ERROR: Preparation failed: failed to pull image "harbor1.mydomain.com/dh/library/swift:focal" with specified policies [always]: Error: No such image: harbor1.mydomain.com/dh/library/swift:focal (manager.go:239:0s)
Will be retried in 3s ...
...

Environment description

Running on Rocky Linux 8 (SELinux enforced). Podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7

Built:      Sun May 15 18:45:11 2022
OS/Arch:    linux/amd64
config.toml contents 
concurrent = 4
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner1.mydomain.net"
  url = "https://gitlab.mydomain.com:4434"
  token = "xxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    host = "unix:///run/user/1000/podman/podman.sock"
    tls_verify = false
    image = "harbor1.mydomain.com/dh/library/alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Used GitLab Runner version

FIPS mode enabled. Using BoringSSL.
Version:      15.1.0
Git revision: 76984217
Git branch:   15-1-stable
GO version:   go1.17.7
Built:        2022-06-20T10:08:34+0000
OS/Arch:      linux/amd64
Edited by Leo