Kubernetes: prepare environment: setting up trapping scripts on emptyDir: error dialing backend: remote error: tls: internal error.
Summary
ERROR: Job failed (system failure): prepare environment: setting up trapping scripts on emptyDir: error dialing backend: remote error: tls: internal error. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
Steps to reproduce
Happens time to time on Kubernetes runner. After restart it is working.
.gitlab-ci.yml
test:core:functional:
rules:
- - if: $CI_PIPELINE_SOURCE == "pipeline"
when: never
- - if: $SKIP_TEST == "true"
when: never
- changes:
- src/**/*
- app/**/*
- apps/**/*
- config/**/*
- cakephp/**/*
- tests/**/*
- migrations/**/*
- database/**/*
- resources/**/*
- routes/**/*
- bundles/**/*
- composer.*
- package.*
- yarn.lock
- pnpm-lock.yaml
- Pipfile*
when: on_success
allow_failure: false
- when: manual
allow_failure: true
cache:
unprotect: true
key:
files:
- composer.lock
prefix: php-
paths:
- ".cache_composer/"
- vendor/
- bin/.phpunit/
- auth.json
extends: ".test"
stage: test
interruptible: true
dependencies: []
image:
name: "${IMAGE_BASE_PATCH}:${IMAGE_VERSION}-test"
coverage: "/^\\s*Lines:\\s*\\d+.\\d+\\%/"
artifacts:
expire_in: 1 week
paths:
- ci
reports:
junit:
- ci/junitreport_func.xml
coverage_report:
coverage_format: cobertura
path: ci/cobertura-coverage-func.xml
before_script:
- php -v
- mkdir -p ci
- - echo "{\"gitlab-token\":{\"gitlab.com\":{\"username\":\"composer_packages\",\"token\":\"$COMPOSER_GITLAB_REGISTRY_TOKEN\"}}}"
> auth.json
- composer global config allow-plugins.isaac/composer-velocita true -q
- composer global require isaac/composer-velocita -q
- composer velocita:enable https://velocita.stage.gotphoto.io/
variables:
APP_ENV: ci
script:
- composer --no-ansi --no-interaction --prefer-dist --optimize-autoloader --no-scripts
install
- php -dextension=pcov -dpcov.enabled=1 -dpcov.directory=. -dpcov.exclude="~vendor~"
./vendor/bin/phpunit --testsuite Functional --coverage-clover="./ci/coverage-clover/Functional.xml"
--coverage-cobertura="./ci/cobertura-coverage-func.xml" --log-junit="ci/junitreport_func.xml"Actual behavior
Jod does not start.
Expected behavior
Job is running
Relevant logs and/or screenshots
From gitlab-runner
Submitting job to coordinator...accepted, but not yet completed bytesize=2562 checksum=crc32:56d7de6b code=202 job=5740485761 job-status= runner=_zYc_Cp1y update-interval=1s
ERROR: Job failed (system failure): prepare environment: setting up trapping scripts on emptyDir: error dialing backend: remote error: tls: internal error. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information duration_s=23.695913709 job=5740485749 project=26949392 runner=_zYc_Cp1y
job log
Running with gitlab-runner 16.6.1 (f5da3c5a)
on gitlab-runner-5b9454c8c4-tcj7r _zYc_Cp1y, system ID: r_qt9uSvozp7Je
feature flags: FF_USE_FASTZIP:true, FF_USE_NEW_BASH_EVAL_STRATEGY:true, FF_SCRIPT_SECTIONS:true, FF_USE_NEW_SHELL_ESCAPE:true
Resolving secrets 00:00
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: gitlab-runner-small
Using Kubernetes executor with image registry.gitlab.com/gotphoto/platform/core/app:8.2-apache-bullseye_54-test ...
Using attach strategy to execute scripts...
Preparing environment 00:24
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 30m0s...
Waiting for pod gitlab-runner-small/runner-zyccp1y-project-26949392-concurrent-36-e8wtvug6 to be running, status is Pending
Unschedulable: "0/10 nodes are available: 1 Insufficient memory, 1 Too many pods, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 1 node(s) had untolerated taint {node_group: arm}, 6 Insufficient cpu. preemption: 0/10 nodes are available: 2 Preemption is not helpful for scheduling, 8 No preemption victims found for incoming pod.."
Waiting for pod gitlab-runner-small/runner-zyccp1y-project-26949392-concurrent-36-e8wtvug6 to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner-small/runner-zyccp1y-project-26949392-concurrent-36-e8wtvug6 to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner-small/runner-zyccp1y-project-26949392-concurrent-36-e8wtvug6 to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner-small/runner-zyccp1y-project-26949392-concurrent-36-e8wtvug6 to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner-small/runner-zyccp1y-project-26949392-concurrent-36-e8wtvug6 to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
ERROR: Job failed (system failure): prepare environment: setting up trapping scripts on emptyDir: error dialing backend: remote error: tls: internal error. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more informationEnvironment description
gitlab Kubernetes runner 16.6.1
config.toml contents
[[runners]]
name = "small"
[runners.feature_flags]
FF_USE_NEW_BASH_EVAL_STRATEGY = true
FF_SCRIPT_SECTIONS = true
FF_USE_NEW_SHELL_ESCAPE = true
FF_USE_POD_ACTIVE_DEADLINE_SECONDS = true
[runners.cache]
Type = "s3"
Shared = true
#5Gi
MaxUploadedArchiveSize = 5368709120
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
AuthenticationType = "access-key"
BucketName = "${s3_cache_bucket_name}"
BucketLocation = "${s3_region}"
Insecure = false
[runners.kubernetes]
image = "scratch"
poll_timeout = ${poll_timeout}
priority_class_name = "${priority_class_name}"
allowed_pull_policies = ["always", "if-not-present", "never"]
pull_policy = ["if-not-present", "always", "always"]
service_account_overwrite_allowed = ""
cpu_request = "900m"
cpu_limit = "2000m"
memory_request = "3Gi"
memory_limit = "6Gi"
service_cpu_request = "100m"
service_cpu_request_overwrite_max_allowed = "3000m"
service_cpu_limit = "2000m"
service_cpu_limit_overwrite_max_allowed = "4000m"
service_memory_request = "512Mi"
service_memory_limit = "2Gi"
helper_cpu_request = "100m"
helper_cpu_request_overwrite_max_allowed = "1000m"
helper_cpu_limit = "1000m"
helper_cpu_limit_overwrite_max_allowed = "6000m"
helper_memory_request = "128Mi"
helper_memory_request_overwrite_max_allowed = "1Gi"
helper_memory_limit = "512Mi"
helper_memory_limit_overwrite_max_allowed = "2Gi"
[[runners.kubernetes.volumes.config_map]]
name = "${config_map_name}"
mount_path = "/tools"
[runners.kubernetes.pod_annotations]
"cluster-autoscaler.kubernetes.io/safe-to-evict" = "false"
"karpenter.sh/do-not-disrupt" = "true"
[runners.kubernetes.pod_labels]
"cluster-autoscaler.kubernetes.io/safe-to-evict" = "false"
"karpenter.sh/do-not-disrupt" = "true"
[runners.kubernetes.affinity]
[runners.kubernetes.affinity.pod_affinity]
[[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution]]
weight = 1
[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term]
topology_key = "kubernetes.io/arch"
namespaces = [%{ for namespace in namespaces ~}"${namespace}",%{~ endfor ~}]
[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector]
[[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector.match_expressions]]
key = "karpenter.sh/do-not-disrupt"
operator = "In"
values = ["true"]Used GitLab Runner version
Kubernetes
Running with gitlab-runner 16.6.1 (f5da3c5a)Possible fixes
- [Placeholder]
Edited by Darren Eastman