Cannot transition status via :enqueue_preparing from :preparing
Summary
When trying to use the connected (integrated) Kubernetes cluster within the CI, we are greeted with the following error: This job failed because the necessary resources were not successfully created.
However, the integration is set up correctly.
This is checked by getting the cluster_status.json via https://gitlab/admin/clusters/6/cluster_status.json, which says connected
.
The GitLab docker container runs behind a corporate proxy. However, the proxy is not required between GitLab and the K8s cluster. Thus, I disabled setting git env
var for gitlab-rails containing http_proxy settings. This had to be removed, since the Kubernetes integration does not honor a no_proxy
setting. Therefore, no proxy is used.
Steps to reproduce
- Integrate a Kubernetes cluster
- Create CI Job, which uses a Kubernetes Envirionment.
What is the current bug behavior?
The CI job does not start.
What is the expected correct behavior?
GitLab should create a namespace and service account and afterwards start the CI jobs.
Relevant logs and/or screenshots
The kubernetes.log
file is not being appended with logs when triggering the non-working CI job.
This is being logged to /var/log/gitlab/gitlab-rails/exceptions.log:
{
"severity":"ERROR",
"time":"2020-09-16T13:38:57.141Z",
"correlation_id":"zB69mfFUSi6",
"extra.sidekiq":{
"class":"Ci::BuildPrepareWorker",
"args":[
"1748"
],
"retry":3,
"queue":"pipeline_processing:ci_build_prepare",
"version":0,
"queue_namespace":"pipeline_processing",
"jid":"f9e9dcbce366cbe8dc13474e",
"created_at":1600263537.0462546,
"meta.user":"Jannis.Lehmann",
"meta.project":"a/b",
"meta.root_namespace":"a",
"meta.caller_id":"Projects::JobsController#retry",
"correlation_id":"zB69mfFUSi6",
"enqueued_at":1600263537.0517967
},
"extra.build_id":1748,
"exception.class":"StateMachines::InvalidTransition",
"exception.message":"Cannot transition status via :enqueue_preparing from :preparing (Reason(s): Transition halted)",
"exception.backtrace":[
"app/services/ci/prepare_build_service.rb:14:in `execute'",
"app/workers/ci/build_prepare_worker.rb:13:in `block in perform'",
"app/workers/ci/build_prepare_worker.rb:12:in `perform'",
"lib/gitlab/metrics/sidekiq_middleware.rb:18:in `block in call'",
"lib/gitlab/metrics/transaction.rb:61:in `run'",
"lib/gitlab/metrics/sidekiq_middleware.rb:18:in `call'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:32:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:40:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:16:in `call'",
"lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
"lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/admin_mode/server.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
"lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
"lib/gitlab/with_request_store.rb:10:in `with_request_store'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/memory_killer.rb:20:in `call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:35:in `call'",
"lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call'",
"lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'",
"lib/gitlab/sidekiq_middleware/monitor.rb:7:in `call'",
"lib/gitlab/sidekiq_logging/structured_logger.rb:18:in `call'"
]
}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
root@git:/# gitlab-rake gitlab:env:info System information System: Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.28.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.3.5 Revision: 467cb4161ad Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https://git. HTTP Clone URL: https://git./some-group/some-project.git SSH Clone URL: ssh://git@git.:123/some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.6.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
root@git:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... Server: ldapmain not verifying SSL hostname of LDAPS server 'private' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found x users of 100 limit. Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/2 ... yes 5/3 ... yes 4/4 ... yes 3/5 ... yes 5/7 ... yes 5/8 ... yes 3/9 ... yes 9/10 ... yes 4/11 ... yes 9/12 ... yes 9/13 ... yes 14/14 ... yes 14/15 ... yes 14/16 ... yes 14/17 ... yes 16/18 ... yes 16/19 ... yes 5/20 ... yes 4/21 ... yes 16/22 ... yes 20/23 ... yes 24/25 ... yes 5/34 ... yes 27/35 ... yes 31/38 ... yes 5/40 ... yes 18/41 ... yes 42/43 ... yes 42/44 ... yes 42/45 ... yes 42/46 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 33 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
If there is anything that I forgot or that could help to find the root cause, please tell me! I'd be happy to assist as I am trying to use the GitLab CI for my bachelor thesis.