unable to upgrade connection: container not found ("build") since 16.6.0
## Summary
<!--
Briefly summarize the bug
-->
The job below fails on GitLab runner versions 16.6.1 and 16.6.0 but works on 16.5.0.
## Steps to reproduce
<!--
What do you need to do to reproduce the bug? Please include job definitions or git repository structure if relevant
-->
1. Create a repository that has this job included in its CI/CD.
2. Commit to the repository to run the pipeline
3. The pipeline should fail.
One interesting thing to note is that we use this job as a downstream pipeline, so it is triggered from another repository.
<!--
Please add the definition of the job from `.gitlab-ci.yml` that is failing
inside of the code blocks (```) below.
-->
<details>
<summary> .gitlab-ci.yml </summary>
```yml
edit-commit-sha:
image:
name: alpine/k8s:1.28.1
entrypoint:
- "/bin/bash"
stage: gitops
resource_group: no_conflicts
rules:
- if: $CI_PIPELINE_SOURCE == "pipeline"
before_script:
- git clone "https://oauth2:${GITLAB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
checkout
- cd checkout
- git checkout $CI_COMMIT_REF_NAME
- git config user.name "${GIT_USER_NAME:-$GITLAB_USER_NAME}"
- git config user.email "${GIT_USER_EMAIL:-$GITLAB_USER_EMAIL}"
script:
- echo ${UPSTREAM_SCRIPT}
- bash -c "${UPSTREAM_SCRIPT}"
after_script:
- cd checkout
- git add .
- echo git commit -m "Updated ${UPSTREAM_PROJECT_NAME}/${UPSTREAM_REFNAME} to ${UPSTREAM_COMMIT_SHA}"
&& echo git push -o ci.skip origin "${CI_COMMIT_REF_NAME}"
- git commit -m "Updated ${UPSTREAM_PROJECT_NAME}/${UPSTREAM_REFNAME} to ${UPSTREAM_COMMIT_SHA}"
&& git push -o ci.skip origin "${CI_COMMIT_REF_NAME}"
```
</details>
## Actual behavior
The gitlab runners cannot initialise the environment and the job fails.
## Expected behavior
<!--
What you should see instead
-->
The job should be able to initialise and start the CI/CD job.
## Relevant logs and/or screenshots
<!--
Paste the job logs inside of the code blocks (```) below so it would be
easier to read.
-->
<details>
<summary> job log </summary>
```sh
Running with gitlab-runner 16.6.1 (f5da3c5a)
on researchable-k8s-runners gJ_Nnoyx, system ID: r_NCY1z8O3lI9H
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image alpine/k8s:1.28.1 ...
Using attach strategy to execute scripts...
Preparing environment
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab-runner/runner-gjnnoyx-project-44350580-concurrent-0-ozx0bj3l to be running, status is Pending
ERROR: Job failed (system failure): prepare environment: setting up trapping scripts on emptyDir: unable to upgrade connection: container not found ("build"). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
```
</details>
## Environment description
<!--
Are you using shared Runners on GitLab.com? Or is it a custom installation?
Which executors are used? Please also provide the versions of related tools
like `docker info` if you are using the Docker executor.
-->
We are running the gitlab runners within Kubernetes on version 1.28.
<!--
Please add the contents of `config.toml` inside of the code blocks (```)
below, remember to remove any secret tokens!
-->
<details>
<summary> config.toml contents </summary>
```toml
[[runners]]
executor = "kubernetes"
environment = [
"DOCKER_HOST=tcp://localhost:2375",
"DOCKER_DRIVER=overlay2",
"DOCKER_TLS_CERTDIR=",
]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu"
serviceAccount= "researchable-shared-runners-gitlab-runner"
pull_policy = ["if-not-present"]
image_pull_secrets = ["gitlab-researchable-image-pull-secret", "docker-researchableuser-image-pull-secret"]
privileged = true
tls_verify = false
cpu_request = "0.5"
cpu_limit = "2"
cpu_request_overwrite_max_allowed = "3"
cpu_limit_overwrite_max_allowed = "4"
memory_request = "512Mi"
memory_limit = "6Gi"
memory_request_overwrite_max_allowed = "8Gi"
memory_limit_overwrite_max_allowed = "8Gi"
service_cpu_request = "0.25"
service_cpu_request_overwrite_max_allowed = "3"
service_cpu_limit_overwrite_max_allowed = "4"
service_memory_request = "512Mi"
helper_image_flavor = "ubuntu"
helper_cpu_request = "0.25"
helper_cpu_request_overwrite_max_allowed = "3"
helper_cpu_limit_overwrite_max_allowed = "4"
helper_memory_request = "256Mi"
helper_memory_request_overwrite_max_allowed = "2Gi"
helper_memory_limit_overwrite_max_allowed = "4Gi"
poll_timeout = 3600
dns_policy = "cluster-first"
cleanup_grace_period_seconds = 60
pod_termination_grace_period_seconds = 5400
resource_availability_check_max_attempts = 30
[runners.cache]
Type = "s3"
Path = "runner"
Shared = true
[runners.cache.s3]
ServerAddress = "ams3.digitaloceanspaces.com"
AccessKey = "TF_VAR_ACCESS_KEY"
SecretKey = "TF_VAR_SECRET_KEY"
BucketName = "k8s-runner-cache"
BucketLocation = "ams-3"
Insecure = false
AuthenticationType = "access-key"
[[runners.kubernetes.volumes.empty_dir]]
name = "docker-certs"
mount_path = "/certs/client"
medium = "Memory"
size_limit = "2Gi"
[runners.kubernetes.pod_annotations]
"pod-cleanup.gitlab.com/ttl" = "2h"
```
</details>
### Used GitLab Runner version
<!--
Please run and paste the output of `gitlab-runner --version`. If you are using
a Runner where you don't have access to, please paste at least the first lines
the from build log, like:
```
Running with gitlab-runner 16.6.1 (f5da3c5a)
on researchable-k8s-runners gJ_Nnoyx, system ID: r_NCY1z8O3lI9H
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image alpine/k8s:1.28.1 ...
```
-->
## Possible fixes
It's possibly related to us setting the entrypoint to `/bin/bash`.
issue