Cannot open or list failed build
Summary
A docker build failed due to me not updating runner configuration to spawn privileged containers. Now any attempt to view a page for the job or pipeline returns a 404 or 500.
Steps to reproduce
- Setup dockerized gitlab runner but don't edit config file to enable privileged access.
- Attempt build from nearly empty repository (just a readme, a basic dockerfile, and the docker .gitlab-ci.yml template).
- Watch in confusion as the build fails and you can no longer list jobs or even access the build.
Example Project
No project.
What is the current bug behavior?
404 and 500 errors returned when the job is retrieved from the database.
What is the expected correct behavior?
The miserable failure of a build log that should be there in a normal GitLab page.
Relevant logs and/or screenshots
Started GET "/root/build-docker/builds/1" for 10.42.40.92 at 2017-06-11 03:11:07 +0000
Processing by Projects::BuildsController#show as HTML
Parameters: {"namespace_id"=>"root", "project_id"=>"build-docker", "id"=>"1"}
Completed 500 Internal Server Error in 141ms (ActiveRecord: 49.8ms)
Module::DelegationError (CommitStatus#sha delegated to pipeline.sha, but pipeline is nil: #<Ci::Build id: 1, status: "failed", finished_at: "2017-06-11 03:09:10", trace: nil, created_at: "2017-06-11 03:08:28", updated_at: "2017-06-11 03:09:10", started_at: "2017-06-11 03:08:32", runner_id: 1, coverage: nil, commit_id: 1, commands: "export IMAGE_TAG=$(echo -en $CI_COMMIT_REF_NAME | ...", name: "build", options: {:image=>"docker:latest", :services=>["docker:dind"]}, allow_failure: false, stage: "build", trigger_request_id: nil, stage_idx: 0, tag: false, ref: "master", user_id: 1, type: "Ci::Build", target_url: nil, description: nil, artifacts_file: nil, project_id: 36, artifacts_metadata: nil, erased_by_id: nil, erased_at: nil, artifacts_expire_at: nil, environment: nil, artifacts_size: nil, when: "on_success", yaml_variables: [], queued_at: "2017-06-11 03:08:29", token: "zCE4pq4j1nkzaSDznCiZ", lock_version: 3, coverage_regex: nil, auto_canceled_by_id: nil, retried: nil>):
app/models/commit_status.rb:14:in `rescue in sha'
app/models/commit_status.rb:14:in `sha'
app/controllers/projects/builds_controller.rb:45:in `show'
app/controllers/application_controller.rb:281:in `set_locale'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
lib/gitlab/etag_caching/middleware.rb:10:in `call'
lib/gitlab/request_context.rb:18:in `call'
...
Started GET "/root/build-docker/pipelines/1" for 10.42.224.103 at 2017-06-11 03:11:32 +0000
Processing by Projects::PipelinesController#show as HTML
Parameters: {"namespace_id"=>"root", "project_id"=>"build-docker", "id"=>"1"}
ActiveRecord::RecordNotFound (Couldn't find Ci::Pipeline with [WHERE "ci_pipelines"."project_id" = ? AND "ci_pipelines"."id" = ?]):
app/controllers/projects/pipelines_controller.rb:157:in `pipeline'
app/controllers/application_controller.rb:281:in `set_locale'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
lib/gitlab/etag_caching/middleware.rb:10:in `call'
lib/gitlab/request_context.rb:18:in `call'
Completed 404 Not Found in 436ms (Views: 311.9ms | ActiveRecord: 47.5ms)
...
Started GET "/root/build-docker/builds" for 10.42.40.92 at 2017-06-11 03:25:32 +0000
Processing by Projects::BuildsController#index as HTML
Parameters: {"namespace_id"=>"root", "project_id"=>"build-docker"}
Completed 500 Internal Server Error in 608ms (ActiveRecord: 57.6ms)
ActionView::Template::Error (CommitStatus#short_sha delegated to pipeline.short_sha, but pipeline is nil: #<Ci::Build id: 1, status: "failed", finished_at: "2017-06-11 03:09:10", trace: nil, created_at: "2017-06-11 03:08:28", updated_at: "2017-06-11 03:09:10", started_at: "2017-06-11 03:08:32", runner_id: 1, coverage: nil, commit_id: 1, commands: "export IMAGE_TAG=$(echo -en $CI_COMMIT_REF_NAME | ...", name: "build", options: {:image=>"docker:latest", :services=>["docker:dind"]}, allow_failure: false, stage: "build", trigger_request_id: nil, stage_idx: 0, tag: false, ref: "master", user_id: 1, type: "Ci::Build", target_url: nil, description: nil, artifacts_file: nil, project_id: 36, artifacts_metadata: nil, erased_by_id: nil, erased_at: nil, artifacts_expire_at: nil, environment: nil, artifacts_size: nil, when: "on_success", yaml_variables: [], queued_at: "2017-06-11 03:08:29", token: "zCE4pq4j1nkzaSDznCiZ", lock_version: 3, coverage_regex: nil, auto_canceled_by_id: nil, retried: nil>):
30: = custom_icon("icon_commit")
31:
32: - if commit_sha
33: = link_to job.short_sha, namespace_project_commit_path(job.project.namespace, job.project, job.sha), class: "commit-id monospace"
34:
35: - if job.stuck?
36: = icon('warning', class: 'text-warning has-tooltip', title: 'Job is stuck. Check runners.')
app/models/commit_status.rb:14:in `rescue in short_sha'
app/models/commit_status.rb:14:in `short_sha'
app/views/projects/ci/builds/_build.html.haml:33:in `_app_views_projects_ci_builds__build_html_haml___2492895015742504954_70319021027300'
app/views/projects/builds/_table.html.haml:23:in `_app_views_projects_builds__table_html_haml__807833131884845598_70319018719280'
app/views/projects/builds/index.html.haml:23:in `_app_views_projects_builds_index_html_haml__3338110324350848802_70319017526000'
app/controllers/application_controller.rb:281:in `set_locale'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
lib/gitlab/etag_caching/middleware.rb:10:in `call'
lib/gitlab/request_context.rb:18:in `call'Output of checks
No check on repository run.
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.11.1
Sidekiq Version:5.0.0
GitLab information
Version: 9.2.5
Revision: 6f2e590
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://******
HTTP Clone URL: https://******/some-group/some-project.git
SSH Clone URL: [******]:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 5.0.4
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/gitResults of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 5.0.4 ? ... OK (5.0.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/11 ... ok
2/12 ... ok
2/13 ... ok
2/14 ... ok
3/15 ... ok
3/16 ... ok
3/17 ... ok
3/18 ... ok
5/19 ... repository is empty
5/20 ... repository is empty
5/21 ... repository is empty
5/22 ... repository is empty
5/23 ... repository is empty
5/24 ... repository is empty
5/25 ... repository is empty
5/26 ... repository is empty
5/27 ... repository is empty
5/28 ... repository is empty
5/29 ... repository is empty
6/30 ... ok
6/33 ... ok
6/34 ... ok
6/35 ... ok
1/36 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
2/11 ... yes
2/12 ... yes
2/13 ... yes
2/14 ... yes
3/15 ... yes
3/16 ... yes
3/17 ... yes
3/18 ... yes
5/19 ... yes
5/20 ... yes
5/21 ... yes
5/22 ... yes
5/23 ... yes
5/24 ... yes
5/25 ... yes
5/26 ... yes
5/27 ... yes
5/28 ... yes
5/29 ... yes
6/30 ... yes
6/33 ... yes
6/34 ... yes
6/35 ... yes
1/36 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.1)
Active users: 1
Checking GitLab ... FinishedPossible fixes
Deleting the job from the database? May do the trick.