HTTP 500 on Pipeline page when Artifact object storage is enabled

Summary

Gitlab returns a HTTP 500 when visiting a pipeline page (https://gitlab.example.com/andy/incubator/56) Error message indicates "Object storage is not enabled" even though it is enabled.

Note this appears to be similar to https://gitlab.com/gitlab-org/gitlab-ee/issues/4532 but our gitlab server configuration explicitly has object storage enabled.

Steps to reproduce

Note: these are the steps I performed to configure artifact storage, I have not tried to reproduce on a second machine.

  1. Enable Enterprise Edition (trial in my case) on gitlab-ee 10.5.1-ee omnibus

  2. Configure S3-backed object storage without credentials (I assumed gitlab could use the IAM role from EC2 metadata). Relevant part of /etc/gitlab/gitlab.rb:

    ### Job Artifacts
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true # EE only
gitlab_rails['artifacts_object_store_remote_directory'] = "REDACTED_BUCKET_NAME"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'AWS',
  'region' => 'us-east-1',
}

  3. Run gitlab-ctl reconfigure then gitlab-rake gitlab:artifacts:migrate and encounter https://gitlab.com/gitlab-org/gitlab-ee/issues/5084.

  4. Reconfigure S3-backed object storage with AWS credentials. Relevant part of /etc/gitlab/gitlab.rb:

    ### Job Artifacts
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true # EE only
gitlab_rails['artifacts_object_store_remote_directory'] = "REDACTED_BUCKET_NAME"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'AWS',
  'region' => 'us-east-1',
  'aws_access_key_id' => 'REDACTED',
  'aws_secret_access_key' => 'REDACTED',
}

  5. Re-run gitlab-ctl reconfigure then gitlab-rake gitlab:artifacts:migrate and observe successful upload of previous build artifacts to your S3 bucket.

  6. Visit page for any pipeline containing a job with build artifacts

Example Project

An example project isn't possible since this is an issue with artifact storage on gitlab-ee, not an issue with a specific project. This problem occurs with all projects containing build artifacts in their pipeline jobs.

What is the current bug behavior?

Visiting a pipeline page yields "500 Whoops, something went wrong on our end." See relevant logs for rails traceback.

What is the expected correct behavior?

Visiting a pipeline page displays the details of the pipeline not an HTTP 500 error.

Relevant logs and/or screenshots

Visiting /andy/incubator/pipelines/68

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/andy/incubator/pipelines/68" for 172.30.1.43 at 2018-03-01 08:59:42 +0000
Processing by Projects::PipelinesController#show as HTML
  Parameters: {"namespace_id"=>"andy", "project_id"=>"incubator", "id"=>"68"}                                                                        
Completed 500 Internal Server Error in 71ms (ActiveRecord: 24.3ms | Elasticsearch: 0.0ms)
 
ActionView::Template::Error (Object Storage is not enabled):
    94:
    95:   %td
    96:     .pull-right
    97:       - if can?(current_user, :read_build, job) && job.artifacts?
    98:         = link_to download_project_job_artifacts_path(job.project, job), rel: 'nofollow', download: '', title: 'Download artifacts', class: 'btn btn-build' do
    99:           = sprite_icon('download')
   100:       - if can?(current_user, :update_build, job)
  app/models/concerns/artifact_migratable.rb:6:in `artifacts_file'
 app/models/concerns/artifact_migratable.rb:14:in `artifacts?'
 app/views/projects/ci/builds/_build.html.haml:97:in `_app_views_projects_ci_builds__build_html_haml__866604242623499455_69897816977080'
  app/views/projects/stage/_stage.html.haml:9:in `_app_views_projects_stage__stage_html_haml___4577791137740978312_69897813399620'
 app/views/projects/pipelines/_with_tabs.html.haml:45:in `_app_views_projects_pipelines__with_tabs_html_haml___2531409917059104737_69897807978520'
 app/views/projects/pipelines/show.html.haml:10:in `_app_views_projects_pipelines_show_html_haml___1034288485960003616_69897891594160'
  app/controllers/projects/pipelines_controller.rb:73:in `show'
 lib/gitlab/i18n.rb:47:in `with_locale'
 lib/gitlab/i18n.rb:53:in `with_user_locale'
  app/controllers/application_controller.rb:333:in `set_locale'
 lib/gitlab/middleware/multipart.rb:95:in `call'
 lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
 lib/gitlab/etag_caching/middleware.rb:11:in `call'
 lib/gitlab/middleware/read_only.rb:31:in `call'
  lib/gitlab/request_context.rb:18:in `call'
 lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'

Visiting /admin/jobs

Started GET "/admin/jobs" for 172.30.1.43 at 2018-03-01 10:18:59 +0000                                
Processing by Admin::JobsController#index as HTML                                     
Completed 500 Internal Server Error in 255ms (ActiveRecord: 64.2ms | Elasticsearch: 0.0ms)
                                               
ActionView::Template::Error (Object Storage is not enabled):                                                                                                                                                           94:                                          
    95:   %td
    96:     .pull-right                                 
    97:       - if can?(current_user, :read_build, job) && job.artifacts?                                                                                                                                              98:         = link_to download_project_job_artifacts_path(job.project, job), rel: 'nofollow', download: '', title: 'Download artifacts', class: 'btn btn-build' do
    99:           = sprite_icon('download')
   100:       - if can?(current_user, :update_build, job)
  app/models/concerns/artifact_migratable.rb:6:in `artifacts_file' 
  app/models/concerns/artifact_migratable.rb:14:in `artifacts?'
  app/views/projects/ci/builds/_build.html.haml:97:in `_app_views_projects_ci_builds__build_html_haml__866604242623499455_69897854010300'
  app/views/projects/jobs/_table.html.haml:23:in `_app_views_projects_jobs__table_html_haml___2681063160359219559_69897852970740'
  app/views/admin/jobs/index.html.haml:23:in `_app_views_admin_jobs_index_html_haml__442535822900753815_69897850258240'
  lib/gitlab/i18n.rb:47:in `with_locale'                                                                                                                                                                             lib/gitlab/i18n.rb:53:in `with_user_locale'
  app/controllers/application_controller.rb:333:in `set_locale'
  lib/gitlab/middleware/multipart.rb:95:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'                                                                                              
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/read_only.rb:31:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Debian 9.3
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.3.6p384
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.3.0
Redis Version:  3.2.11
Git Version:    2.14.3
Sidekiq Version:5.0.5
Go Version:     unknown

GitLab information
Version:        10.5.1-ee
Revision:       f1e14d0
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.6
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2


GitLab Shell
Version:        6.0.3
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...

GitLab Shell version >= 6.0.3 ? ... OK (6.0.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/2 ... ok
4/3 ... ok
6/4 ... ok
6/5 ... ok
6/6 ... ok
6/7 ... ok
6/8 ... ok
6/9 ... ok
4/10 ... ok
6/11 ... ok
4/12 ... ok
4/13 ... ok
4/14 ... ok
4/15 ... ok
4/16 ... ok
6/17 ... ok
6/19 ... ok
6/20 ... ok
6/21 ... ok
6/22 ... ok
6/23 ... ok
6/24 ... ok
6/25 ... ok
6/26 ... ok
6/27 ... ok
6/28 ... ok
6/29 ... ok
6/30 ... ok
6/31 ... ok
6/32 ... ok
6/33 ... ok
6/34 ... ok
6/35 ... ok
6/36 ... ok
6/37 ... ok
6/38 ... ok
6/39 ... ok
6/40 ... ok
6/41 ... ok
6/42 ... ok
6/43 ... ok
6/44 ... ok
6/45 ... ok
6/46 ... ok
4/47 ... ok
4/48 ... ok
6/51 ... ok
2/52 ... ok
5/53 ... ok
10/55 ... ok
5/56 ... ok
5/57 ... ok
5/58 ... ok
5/59 ... ok
2/60 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/2 ... yes
4/3 ... yes
6/4 ... yes
6/5 ... yes
6/6 ... yes
6/7 ... yes
6/8 ... yes
6/9 ... yes
4/10 ... yes
6/11 ... yes
4/12 ... yes
4/13 ... yes
4/14 ... yes
4/15 ... yes
4/16 ... yes
6/17 ... yes
6/19 ... yes
6/20 ... yes
6/21 ... yes
6/22 ... yes
6/23 ... yes
6/24 ... yes
6/25 ... yes
6/26 ... yes
6/27 ... yes
6/28 ... yes
6/29 ... yes
6/30 ... yes
6/31 ... yes
6/32 ... yes
6/33 ... yes
6/34 ... yes
6/35 ... yes
6/36 ... yes
6/37 ... yes
6/38 ... yes
6/39 ... yes
6/40 ... yes
6/41 ... yes
6/42 ... yes
6/43 ... yes
6/44 ... yes
6/45 ... yes
6/46 ... yes
4/47 ... yes
4/48 ... yes
6/51 ... yes
2/52 ... yes
5/53 ... yes
10/55 ... yes
5/56 ... yes
5/57 ... yes
5/58 ... yes
5/59 ... yes
2/60 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.6)
Git version >= 2.9.5 ? ... yes (2.14.3)
Git user has default SSH configuration? ... yes
Active users: ... 5
Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)


Checking GitLab ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)