Cannot transition status via :enqueue_preparing from :preparing

Summary

When trying to use the connected (integrated) Kubernetes cluster within the CI, we are greeted with the following error: This job failed because the necessary resources were not successfully created. However, the integration is set up correctly. This is checked by getting the cluster_status.json via https://gitlab/admin/clusters/6/cluster_status.json, which says connected.

The GitLab docker container runs behind a corporate proxy. However, the proxy is not required between GitLab and the K8s cluster. Thus, I disabled setting git env var for gitlab-rails containing http_proxy settings. This had to be removed, since the Kubernetes integration does not honor a no_proxy setting. Therefore, no proxy is used.

Steps to reproduce

Integrate a Kubernetes cluster
Create CI Job, which uses a Kubernetes Envirionment.

What is the current bug behavior?

The CI job does not start.

What is the expected correct behavior?

GitLab should create a namespace and service account and afterwards start the CI jobs.

Relevant logs and/or screenshots

The kubernetes.log file is not being appended with logs when triggering the non-working CI job. This is being logged to /var/log/gitlab/gitlab-rails/exceptions.log:

{
   "severity":"ERROR",
   "time":"2020-09-16T13:38:57.141Z",
   "correlation_id":"zB69mfFUSi6",
   "extra.sidekiq":{
      "class":"Ci::BuildPrepareWorker",
      "args":[
         "1748"
      ],
      "retry":3,
      "queue":"pipeline_processing:ci_build_prepare",
      "version":0,
      "queue_namespace":"pipeline_processing",
      "jid":"f9e9dcbce366cbe8dc13474e",
      "created_at":1600263537.0462546,
      "meta.user":"Jannis.Lehmann",
      "meta.project":"a/b",
      "meta.root_namespace":"a",
      "meta.caller_id":"Projects::JobsController#retry",
      "correlation_id":"zB69mfFUSi6",
      "enqueued_at":1600263537.0517967
   },
   "extra.build_id":1748,
   "exception.class":"StateMachines::InvalidTransition",
   "exception.message":"Cannot transition status via :enqueue_preparing from :preparing (Reason(s): Transition halted)",
   "exception.backtrace":[
      "app/services/ci/prepare_build_service.rb:14:in `execute'",
      "app/workers/ci/build_prepare_worker.rb:13:in `block in perform'",
      "app/workers/ci/build_prepare_worker.rb:12:in `perform'",
      "lib/gitlab/metrics/sidekiq_middleware.rb:18:in `block in call'",
      "lib/gitlab/metrics/transaction.rb:61:in `run'",
      "lib/gitlab/metrics/sidekiq_middleware.rb:18:in `call'",
      "lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:32:in `perform'",
      "lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:40:in `perform'",
      "lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
      "lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'",
      "lib/gitlab/sidekiq_middleware/worker_context/server.rb:16:in `call'",
      "lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
      "lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
      "lib/gitlab/sidekiq_middleware/admin_mode/server.rb:8:in `call'",
      "lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in `call'",
      "lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
      "lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
      "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
      "lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
      "lib/gitlab/with_request_store.rb:10:in `with_request_store'",
      "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
      "lib/gitlab/sidekiq_middleware/memory_killer.rb:20:in `call'",
      "lib/gitlab/sidekiq_middleware/server_metrics.rb:35:in `call'",
      "lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call'",
      "lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'",
      "lib/gitlab/sidekiq_middleware/monitor.rb:7:in `call'",
      "lib/gitlab/sidekiq_logging/structured_logger.rb:18:in `call'"
   ]
}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

root@git:/# gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.28.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.3.5
Revision:       467cb4161ad
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.7
URL:            https://git.
HTTP Clone URL: https://git./some-group/some-project.git
SSH Clone URL:  ssh://git@git.:123/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        13.6.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check


root@git:/# gitlab-rake gitlab:check SANITIZE=true     
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'private'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
        User output sanitized. Found x users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... 
yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
4/2 ... yes
5/3 ... yes
4/4 ... yes
3/5 ... yes
5/7 ... yes
5/8 ... yes
3/9 ... yes
9/10 ... yes
4/11 ... yes
9/12 ... yes
9/13 ... yes
14/14 ... yes
14/15 ... yes
14/16 ... yes
14/17 ... yes
16/18 ... yes
16/19 ... yes
5/20 ... yes
4/21 ... yes
16/22 ... yes
20/23 ... yes
24/25 ... yes
5/34 ... yes
27/35 ... yes
31/38 ... yes
5/40 ... yes
18/41 ... yes
42/43 ... yes
42/44 ... yes
42/45 ... yes
42/46 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 33
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
  Try fixing it:
  Please migrate all projects to hashed storage
  as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
  For more information see:
  doc/administration/repository_storage_types.md

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

If there is anything that I forgot or that could help to find the root cause, please tell me! I'd be happy to assist as I am trying to use the GitLab CI for my bachelor thesis.

Edited Sep 16, 2020 by Ghost User