Build container fails in gitlab-runner but works locally via docker run

Summary

Python within a docker image does not work when run via gitlab-runner in a Kubernetes environment, however, works locally via docker run.

cc @jclar

Steps to reproduce

Here is the Dockerfile of the image

FROM gcr.io/kaniko-project/executor:v1.3.0@sha256:b9eec410fa32cd77cdb7685c70f86a96debb8b087e77e63d7fe37eaadb178709

FROM python:3.9-buster as FINAL

COPY --from=0 /kaniko/executor /kaniko/executor
COPY --from=0 /kaniko/docker-credential-gcr /kaniko/docker-credential-gcr
COPY --from=0 /kaniko/docker-credential-ecr-login /kaniko/docker-credential-ecr-login
COPY --from=0 /kaniko/docker-credential-acr /kaniko/docker-credential-acr
COPY --from=0 /kaniko/docker-credential-acr-env /kaniko/docker-credential-acr-env
COPY --from=0 /kaniko/ssl/certs/ /kaniko/ssl/certs/
COPY --from=0 /etc/nsswitch.conf /etc/nsswitch.conf

RUN pip3 install pyyaml requests python-jose docopt
RUN apt-get update && apt-get install --no-install-recommends -y curl wget
ADD build_client.py /opt/scripts/build_client.py

ENV HOME /root
ENV USER /root
ENV PATH "$PATH:/kaniko"
ENV SSL_CERT_DIR=/kaniko/ssl/certs
ENV DOCKER_CONFIG /kaniko/.docker/
ENV DOCKER_CREDENTIAL_GCR_CONFIG /kaniko/.config/gcloud/docker_credential_gcr_config.json
RUN ["docker-credential-gcr", "config", "--token-source=env"]

ENTRYPOINT ["/kaniko/executor"]

Here are the instructions to run the image locally

docker run -it --entrypoint "" gitlab-kaniko-build-docker bash
python3 --version

Here is the yaml to run it inside gitlab runner

.gitlab-ci.yml 
test:
  image:
    name: gitlab-kaniko-build:latest
    entrypoint: [""]
  script:
    - |-
        python3 --version

Actual behavior

python in the image fails when run as a container in gitlab runner via kubernetes executor with the following error:

python3: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

Expected behavior

Image running locally and in gitlab runner are deterministic and python works inside of the image in gitlab runner

Environment description

config.toml contents 
listen_address = ":9252"
concurrent = 10
check_interval = 5
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  request_concurrency = 1
  executor = "kubernetes"
  environment = ["DOCKER_HOST=tcp://localhost:2375", "DOCKER_TLS_CERTDIR="]
  [runners.custom_build_dir]
  [runners.cache]
    Type = "gcs"
    Shared = true
    [runners.cache.s3]
    [runners.cache.gcs]
      CredentialsFile = "/secrets/gcs-application-credentials-file"
      BucketName = "gitlab-builds-cache"
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "alpine"
    namespace = "gitlab-managed-apps"
    namespace_overwrite_allowed = ""
    privileged = true
    cpu_limit = "4000m"
    cpu_request = "512m"
    memory_limit = "16Gi"
    memory_request = "1Gi"
    service_cpu_limit = "500m"
    service_cpu_request = "100m"
    service_memory_limit = "1Gi"
    service_memory_request = "128Mi"
    helper_cpu_request = "100m"
    helper_memory_request = "128Mi"
    poll_timeout = 300
    service_account = "gitlab-runner-ksa"
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.node_selector]
      role = "build"
    [runners.kubernetes.node_tolerations]
      "role=build" = "NoSchedule"
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_labels]
      gitlab-runner-worker = "true"
    [runners.kubernetes.pod_annotations]
      "cluster-autoscaler.kubernetes.io/safe-to-evict" = "false"
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

Used GitLab Runner version

Relevant issues/links/etc.

I think these issues are related, but the solutions/workarounds noted there have not helped me.

Possible fixes

Edited by Yash