Deployment jobs broken as of 11.10.0
Summary
We recently upgraded to 11.10.0. Jobs that generate a deployment started to fail with HTTP errors 500.
No Kubernetes cluster is added or setup, auto-devops is disabled.
Steps to reproduce
.gitlab-ci.yml
deploy:
stage: deploy
tags:
- specific-runner-tag
dependencies: []
environment:
name: production
url: "https://www.example.org/"
when: manual
only:
- master
variables:
GIT_STRATEGY: none
before_script:
# add deploy key
- source ~/.bashrc
- eval $(ssh-agent -s)
- ssh-add <(echo "$DEPLOY_KEY")
script:
- echo "foo"
What is the current bug behavior?
- When triggering the manual deployment job an HTTP error 500 is returned.
- No job is started
What is the expected correct behavior?
Deployment job should start.
Relevant logs and/or screenshots
production.log
Started POST "/<redacted-namespace>/<redacted-project>/-/jobs/93188/play" for <redacted-ip> at 2019-04-23 11:02:22 +0200
Processing by Projects::JobsController#play as HTML
Parameters: {"authenticity_token"=>"[FILTERED]", "namespace_id"=>"<redacted-namespace>", "project_id"=>"<redacted-project>", "id"=>"93188"}
Completed 500 Internal Server Error in 32ms (ActiveRecord: 5.0ms)
Sending event 6c1712bbb3fa4338b28364f90a9b41ad to Sentry
NoMethodError (undefined method `cluster' for #<KubernetesService:0x00007f9365b21f10>):
app/models/deployment.rb:82:in `cluster'
lib/gitlab/metrics/instrumentation.rb:161:in `block in cluster'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `cluster'
lib/gitlab/ci/build/prerequisite/kubernetes_namespace.rb:21:in `deployment_cluster'
lib/gitlab/ci/build/prerequisite/kubernetes_namespace.rb:9:in `unmet?'
lib/gitlab/ci/build/prerequisite/factory.rb:19:in `select'
lib/gitlab/ci/build/prerequisite/factory.rb:19:in `unmet'
app/models/ci/build.rb:388:in `prerequisites'
lib/gitlab/metrics/instrumentation.rb:161:in `block in prerequisites'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `prerequisites'
app/models/ci/build.rb:384:in `any_unmet_prerequisites?'
app/services/ci/play_build_service.rb:12:in `execute'
app/models/ci/build.rb:347:in `play'
app/controllers/projects/jobs_controller.rb:93:in `play'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 8.11 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown
GitLab information Version: 11.10.0 Revision: 8a802d1c6b7 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.11 URL: https:// HTTP Clone URL: https:///some-group/some-project.git SSH Clone URL: git@:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml, gitlab, github, bitbucket
GitLab Shell Version: 9.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.0.0 ? ... OK (9.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/1 ... yes 6/4 ... yes 6/5 ... yes 13/6 ... yes 6/7 ... yes 3/9 ... yes 53/10 ... yes 7/11 ... yes 15/12 ... yes 8/14 ... yes 8/15 ... yes 7/16 ... yes 15/17 ... yes 7/19 ... yes 19/20 ... yes 15/21 ... yes 7/22 ... yes 7/23 ... yes 19/24 ... yes 19/25 ... yes 19/26 ... yes 1/27 ... yes 19/28 ... yes 7/29 ... yes 15/30 ... yes 13/31 ... yes 8/33 ... yes 8/34 ... yes 15/35 ... yes 53/38 ... yes 3/39 ... yes 22/40 ... yes 13/41 ... yes 13/42 ... yes 10/44 ... yes 13/46 ... yes 13/47 ... yes 19/48 ... yes 22/49 ... yes 13/50 ... yes 22/51 ... yes 22/54 ... yes 34/56 ... yes 22/57 ... yes 7/58 ... yes 53/59 ... yes 34/60 ... yes 13/61 ... yes 13/62 ... yes 13/63 ... yes 34/64 ... yes 7/65 ... yes 1/67 ... yes 3/68 ... yes 7/69 ... yes 6/70 ... yes 7/71 ... yes 7/72 ... yes 25/73 ... yes 18/74 ... yes 25/75 ... yes 26/76 ... yes 7/77 ... yes 7/78 ... yes 7/79 ... yes 43/80 ... yes 1/81 ... yes 7/82 ... yes 7/83 ... yes 26/84 ... yes 25/85 ... yes 9/86 ... yes 26/87 ... yes 1/88 ... yes 19/89 ... yes 13/90 ... yes 19/92 ... yes 19/93 ... yes 16/94 ... yes 13/95 ... yes 19/96 ... yes 13/97 ... yes 15/98 ... yes 15/99 ... yes 26/100 ... yes 26/101 ... yes 25/103 ... yes 13/104 ... yes 35/105 ... yes 35/106 ... yes 15/107 ... yes 15/108 ... yes 35/109 ... yes 35/110 ... yes 7/111 ... yes 1/112 ... yes 36/113 ... yes 25/114 ... yes 7/115 ... yes 19/116 ... yes 37/117 ... yes 76/118 ... yes 37/119 ... yes 9/120 ... yes 35/121 ... yes 35/122 ... yes 18/123 ... yes 9/124 ... yes 37/125 ... yes 3/126 ... yes 19/127 ... yes 40/128 ... yes 39/129 ... yes 39/130 ... yes 25/131 ... yes 16/132 ... yes 9/133 ... yes 3/134 ... yes 35/135 ... yes 35/136 ... yes 35/137 ... yes 35/138 ... yes 35/139 ... yes 35/140 ... yes 35/141 ... yes 35/142 ... yes 35/143 ... yes 35/144 ... yes 35/145 ... yes 35/146 ... yes 35/147 ... yes 35/148 ... yes 35/149 ... yes 35/150 ... yes 35/151 ... yes 35/152 ... yes 35/153 ... yes 35/154 ... yes 35/155 ... yes 35/156 ... yes 35/157 ... yes 35/158 ... yes 35/159 ... yes 35/160 ... yes 35/161 ... yes 35/162 ... yes 35/163 ... yes 35/164 ... yes 35/165 ... yes 35/166 ... yes 35/167 ... yes 35/168 ... yes 13/169 ... yes 35/170 ... yes 3/171 ... yes 3/172 ... yes 18/173 ... yes 40/175 ... yes 37/176 ... yes 40/177 ... yes 35/178 ... yes 35/179 ... yes 18/180 ... yes 35/181 ... yes 3/182 ... yes 54/183 ... yes 35/184 ... yes 35/185 ... yes 43/186 ... yes 35/187 ... yes 18/188 ... yes 19/189 ... yes 35/190 ... yes 19/191 ... yes 35/192 ... yes 34/193 ... yes 25/194 ... yes 35/195 ... yes 35/196 ... yes 35/197 ... yes 35/198 ... yes 3/199 ... yes 35/200 ... yes 18/201 ... yes 25/202 ... yes 18/203 ... yes 18/204 ... yes 18/205 ... yes 35/206 ... yes 37/207 ... yes 49/208 ... yes 49/209 ... yes 43/210 ... yes 43/211 ... yes 25/212 ... yes 18/213 ... yes 70/214 ... yes 49/215 ... yes 50/216 ... yes 59/217 ... yes 35/218 ... yes 9/219 ... yes 15/220 ... yes 18/221 ... yes 35/222 ... yes 50/223 ... yes 18/224 ... yes 50/225 ... yes 18/226 ... yes 49/227 ... yes 50/228 ... yes 18/229 ... yes 35/230 ... yes 37/231 ... yes 37/232 ... yes 18/233 ... yes 9/234 ... yes 35/235 ... yes 9/236 ... yes 49/237 ... yes 35/238 ... yes 35/239 ... yes 18/240 ... yes 18/241 ... yes 27/242 ... yes 35/243 ... yes 51/244 ... yes 18/245 ... yes 19/246 ... yes 35/248 ... yes 13/249 ... yes 9/250 ... yes 1/251 ... yes 49/252 ... yes 49/253 ... yes 35/254 ... yes 35/255 ... yes 35/256 ... yes 35/257 ... yes 15/258 ... yes 15/259 ... yes 15/260 ... yes 35/261 ... yes 50/262 ... yes 35/263 ... yes 35/264 ... yes 35/266 ... yes 35/267 ... yes 35/268 ... yes 3/269 ... yes 49/270 ... yes 35/271 ... yes 35/272 ... yes 1/273 ... yes 35/274 ... yes 15/275 ... yes 35/276 ... yes 55/277 ... yes 3/278 ... yes 35/279 ... yes 56/280 ... yes 56/281 ... yes 56/282 ... yes 56/284 ... yes 35/285 ... yes 50/286 ... yes 35/288 ... yes 58/289 ... yes 40/290 ... yes 59/291 ... yes 56/292 ... yes 51/293 ... yes 61/294 ... yes 35/295 ... yes 3/296 ... yes 41/297 ... yes 56/298 ... yes 41/299 ... yes 35/300 ... yes 35/301 ... yes 77/302 ... yes 35/303 ... yes 35/304 ... yes 50/305 ... yes 70/306 ... yes 35/307 ... yes 35/308 ... yes 70/309 ... yes 61/310 ... yes 61/311 ... yes 61/313 ... yes 35/314 ... yes 77/315 ... yes 71/316 ... yes 54/318 ... yes 77/319 ... yes 9/320 ... yes 35/322 ... yes 73/323 ... yes 35/324 ... yes 74/325 ... yes 35/326 ... yes 35/327 ... yes 35/328 ... yes 76/329 ... yes 76/330 ... yes 76/331 ... yes 76/332 ... yes 76/333 ... yes 76/334 ... yes 76/335 ... yes 8/337 ... yes 70/338 ... yes 43/339 ... yes 49/340 ... yes 35/341 ... yes 35/342 ... yes 61/343 ... yes 61/344 ... yes 59/345 ... yes 35/346 ... yes 61/347 ... yes 18/348 ... yes 71/349 ... yes 61/350 ... yes 61/351 ... yes 26/352 ... yes 9/353 ... yes 79/354 ... yes 79/355 ... yes 94/357 ... yes 71/358 ... yes 8/359 ... yes 35/360 ... yes 8/361 ... yes 71/362 ... yes 80/363 ... yes 61/364 ... yes 74/365 ... yes 3/366 ... yes 61/367 ... yes 35/368 ... yes 71/369 ... yes 71/370 ... yes 35/371 ... yes 1/372 ... yes 1/373 ... yes 50/374 ... yes 50/375 ... yes 71/376 ... yes 82/377 ... yes 82/378 ... yes 85/379 ... yes 35/380 ... yes 35/381 ... yes 83/382 ... yes 71/383 ... yes 84/384 ... yes 71/385 ... yes 71/386 ... yes 71/387 ... yes 35/388 ... yes 35/389 ... yes 88/390 ... yes 87/391 ... yes 74/392 ... yes 49/393 ... yes 57/394 ... yes 88/395 ... yes 88/396 ... yes 69/397 ... yes 88/398 ... yes 94/399 ... yes 49/400 ... yes 83/401 ... yes 18/402 ... yes 74/403 ... yes 74/404 ... yes 97/405 ... yes 71/406 ... yes 88/407 ... yes 76/408 ... yes 1/410 ... yes 71/411 ... yes 49/412 ... yes 74/413 ... yes 74/414 ... yes 95/415 ... yes 98/416 ... yes 98/417 ... yes 92/418 ... yes 100/419 ... yes 98/420 ... yes 98/421 ... yes 84/422 ... yes 83/423 ... yes 98/424 ... yes 98/426 ... yes 101/427 ... yes 98/428 ... yes 100/429 ... yes 98/430 ... yes 98/431 ... yes 98/432 ... yes 98/433 ... yes 61/434 ... yes 98/435 ... yes 98/436 ... yes 74/437 ... yes 97/438 ... yes 98/439 ... yes 102/440 ... yes 82/441 ... yes 103/442 ... yes 74/443 ... yes 95/444 ... yes 85/445 ... yes 85/446 ... yes 71/447 ... yes 3/448 ... yes 84/449 ... yes 3/450 ... yes 74/451 ... yes 98/452 ... yes 3/453 ... yes 106/454 ... yes 106/455 ... yes 106/456 ... yes 71/457 ... yes 70/458 ... yes 70/459 ... yes 70/460 ... yes 80/461 ... yes 3/462 ... yes 106/464 ... yes 106/478 ... yes 108/479 ... yes 110/484 ... yes 110/485 ... yes 110/486 ... yes 110/487 ... yes 110/488 ... yes 110/489 ... yes 110/490 ... yes 110/491 ... yes 110/492 ... yes 110/493 ... yes 110/494 ... yes 110/495 ... yes 110/496 ... yes 110/497 ... yes 110/498 ... yes 110/499 ... yes 110/500 ... yes 110/501 ... yes 110/502 ... yes 110/503 ... yes 110/504 ... yes 110/505 ... yes 110/506 ... yes 110/507 ... yes 110/508 ... yes 110/509 ... yes 110/510 ... yes 110/511 ... yes 110/512 ... yes 110/513 ... yes 110/514 ... yes 110/515 ... yes 92/516 ... yes 110/517 ... yes 110/518 ... yes 71/519 ... yes 71/520 ... yes 74/521 ... yes 106/522 ... yes 1/523 ... yes 70/524 ... yes 70/525 ... yes 113/526 ... yes 106/527 ... yes 106/528 ... yes 106/529 ... yes 114/530 ... yes 106/531 ... yes 114/532 ... yes 110/533 ... yes 43/535 ... yes 110/536 ... yes 110/537 ... yes 110/538 ... yes 110/539 ... yes 110/540 ... yes 110/541 ... yes 110/542 ... yes 110/543 ... yes 57/544 ... yes 106/545 ... yes 74/546 ... yes 49/547 ... yes 110/548 ... yes 84/549 ... yes 83/550 ... yes 71/551 ... yes 106/552 ... yes 50/553 ... yes 110/554 ... yes 110/555 ... yes 110/556 ... yes 1/557 ... yes 57/558 ... yes 110/559 ... yes 54/560 ... yes 49/561 ... yes 71/563 ... yes 115/564 ... yes 116/565 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 23
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
- Can be fixed by turning off the feature flag
ci_preparing_state
. - May be related to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25586.