Skip to content

Unable to schedule properly job on EKS-fargate-profile(containerdd:1.4.1)

Summary

Unable to schedule gitlab-runner jobs running on EKS-fargate-profile when source image is based on the alpine

Steps to reproduce

we need to have deployed EKS cluster(1.19). Installed Gitlab-server and deployed gitlab-runner-master on EKS-fargate-profile

.gitlab-ci.yml 
default:
  tags: 
    - eks-nonprod-fg

stages:
  - test


alpine_test: 
  image: XXX.com:8085/alpine/helm:3.2.1
  stage: test
  script:
    - echo "test"

Actual behavior

Jobs is not properly scheduled and Job is not complete properly. During by tests and execution pod directly via kubectl I am able to spin up container properly ( any alpine image)

Expected behavior

Job has to be completed properly. So if a new image should be created (kaniko) then image should be published and builded

Relevant logs and/or screenshots

job log 
Running with gitlab-runner 13.12.0 (7a6612da)
  on gitlab-runner-fg-gitlab-runner-7c85bff8bb-d5b6k Jfv6YfYr
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner-fg
Using Kubernetes executor with image XXX:8085/alpine/helm:3.2.1 ...
Preparing environment
01:22
WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrate-helper-image-to-registrygitlabcom
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner-fg/runner-jfv6yfyr-project-386-concurrent-0sgjkz to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-jfv6yfyr-project-386-concurrent-0sgjkz via gitlab-runner-fg-gitlab-runner-7c85bff8bb-d5b6k...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/XXX/0/XXX/fargate-gitlab-runner/.git/
Created fresh repository.
Checking out 497e247d as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
Cleaning up file based variables
00:00
ERROR: Job failed (system failure): unable to upgrade connection: container not found ("build")

Environment description

I am using kubernetes executor and EKS cluster with version 1.19. Fargate -> eks-fargate-profile is using containerd(1.4.1)

config.toml contents 
listen_address = ":9252"
concurrent = 50
check_interval = 3
log_level = "debug"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "fargate-gitlab-runner-86c4c7c95f-swbwh"
  output_limit = 16384
  request_concurrency = 1
  url = "https://XXX.com/"
  token = "XXX"
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "XXX:8085/ubuntu:16.04"
    namespace = "gitlab-runner-fg"
    namespace_overwrite_allowed = ""
    privileged = false
    cpu_limit = "1500m"
    cpu_request = "1500m"
    memory_limit = "4000Mi"
    pull_policy = [""]
    image_pull_secrets = ["docker-registry-login"]
    poll_timeout = 1200
    service_account = "infra-fargate-gitlab-runner"
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    helper_image = "XXX:8085/gitlab/gitlab-runner-helper:ubuntu-x86_64-c97a73fa"
    dns_policy = cluster-first-with-host-net
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]
    [runners.kubernetes.dns_config]

Used GitLab Runner version

gitlab-runner --version
Version:      13.12.0
Git revision: 7a6612da
Git branch:   13-12-stable
GO version:   go1.13.8
Built:        2021-05-20T15:16:05+0000
OS/Arch:      linux/amd64

Possible fixes

Edited by Tomas Holcek