customers.gitlab.com unavailable (returning 500 errors)

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.

---

Summary

A change was made in https://gitlab.com/gitlab-org/customers-gitlab-com that brought the service down

Service(s) affected : customers.gitlab.com

Team attribution :

Minutes downtime or degradation : 35 min of downtime

Timeline

2019-10-07

• 12:51 UTC - chef-client run started on the customers VM which most likely picks up the commit that results in customers.gitlab.com starting to return 500 errors
• 12:53 UTC - monitoring picks up that customers.gitlab.com is broken
• 12:58 UTC - we get paged
• 13:05 UTC - a number of merge requests with patches reverting the change are created, see:
  • https://gitlab.com/gitlab-org/customers-gitlab-com/pipelines
  • https://gitlab.com/gitlab-org/customers-gitlab-com/merge_requests/517
  • https://gitlab.com/gitlab-org/customers-gitlab-com/merge_requests/516
• 13:26 UTC - a chef-client run is triggered manually on the customers machine to deploy as quickly as possible
• 13:28 UTC - customers.gitlab.com is back online

added incident label

closed

changed the description

changed the description

mentioned in issue gitlab-org/customers-gitlab-com#671 (closed)

mentioned in issue #1226 (closed)

Error occurred while compiling the assets. Unsure why the deployment continued there

Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: ================================================================================
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Error executing action `run` on resource 'execute[precompile:assets]'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: ================================================================================
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Mixlib::ShellOut::ShellCommandFailed
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: ------------------------------------
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Expected process to exit with [0], but received '1'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: ---- Begin output of /usr/local/bin/bundle exec /usr/local/bin/rake assets:precompile RAILS_ENV=production ----
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: STDOUT: `/` is not writable.
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: STDERR: I, [2019-10-07T13:25:15.711930 #128254]  INFO -- : Writing /home/gitlab-customers/customers-gitlab-com/public/assets/vendor/zuora.min-80a55ecea2994d1ff6cb2632f60186fc4efc08aca3d45becf41e309911bcab2f.js
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: I, [2019-10-07T13:25:15.713262 #128254]  INFO -- : Writing /home/gitlab-customers/customers-gitlab-com/public/assets/vendor/zuora.min-80a55ecea2994d1ff6cb2632f60186fc4efc08aca3d45becf41e309911bcab2f.js.gz
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: rake aborted!
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Sass::SyntaxError: File to import not found or unreadable: ../../../node_modules/bootstrap-switch/dist/css/bootstrap3/bootstrap-switch.
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Load paths:
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/app/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/app/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/coffee-rails-4.1.1/lib/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/app/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/app/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/vendor/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/remotipart-1.2.1/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/nested_form-0.3.2/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-ui-rails-5.0.5/app/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-ui-rails-5.0.5/app/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-ui-rails-5.0.5/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-rails-4.1.1/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/font-awesome-rails-4.6.3.1/app/assets/fonts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/font-awesome-rails-4.6.3.1/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/fonts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/vendor/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/app/assets/stylesheets/application.scss:8
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:67:in `rescue in import'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:44:in `import'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:28:in `imported_file'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:37:in `css_import?'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:310:in `visit_import'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `block in visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `block in with_base'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `with_base'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `block in visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `map'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:167:in `block in visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:179:in `with_environment'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:166:in `visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `block in visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:186:in `visit_root'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:157:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:10:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/root_node.rb:36:in `css_tree'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/root_node.rb:20:in `render'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/engine.rb:290:in `render'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-rails-5.0.7/lib/sass/rails/template.rb:47:in `evaluate'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/tilt-2.0.9/lib/tilt/template.rb:109:in `render'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy_tilt_processor.rb:25:in `call'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:75:in `call_processor'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `reverse_each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:134:in `load_from_unloaded'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:60:in `block in load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:44:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:20:in `block in initialize'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:47:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/bundle.rb:23:in `block in call'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/utils.rb:200:in `dfs'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/bundle.rb:24:in `call'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:75:in `call_processor'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `reverse_each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:134:in `load_from_unloaded'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:60:in `block in load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:44:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:20:in `block in initialize'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:47:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/base.rb:66:in `find_asset'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/base.rb:73:in `find_all_linked_assets'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/manifest.rb:142:in `block in find'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:114:in `block (2 levels) in logical_paths'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:228:in `block in stat_tree'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:212:in `block in stat_directory'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:209:in `each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:209:in `stat_directory'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:227:in `stat_tree'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:105:in `each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:105:in `block in logical_paths'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:104:in `each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:104:in `logical_paths'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/manifest.rb:140:in `find'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/manifest.rb:186:in `compile'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/task.rb:68:in `block (3 levels) in define'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/rake/sprocketstask.rb:147:in `with_logger'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/task.rb:67:in `block (2 levels) in define'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /usr/local/bin/bundle:23:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /usr/local/bin/bundle:23:in `<main>'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Sass::SyntaxError: File to import not found or unreadable: ../../../node_modules/bootstrap-switch/dist/css/bootstrap3/bootstrap-switch.
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Load paths:
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/app/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/app/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/coffee-rails-4.1.1/lib/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/app/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/app/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/rails_admin-0.8.1/vendor/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/remotipart-1.2.1/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/nested_form-0.3.2/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-ui-rails-5.0.5/app/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-ui-rails-5.0.5/app/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-ui-rails-5.0.5/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/jquery-rails-4.1.1/vendor/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/font-awesome-rails-4.6.3.1/app/assets/fonts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/font-awesome-rails-4.6.3.1/app/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/javascripts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/fonts
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/bootstrap-sass-3.4.1/assets/images
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]:   /home/gitlab-customers/vendor/assets/stylesheets
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:65:in `import'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:28:in `imported_file'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/import_node.rb:37:in `css_import?'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:310:in `visit_import'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `block in visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `block in with_base'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `with_base'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `block in visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `map'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:167:in `block in visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:179:in `with_environment'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:166:in `visit_children'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `block in visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:186:in `visit_root'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:157:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:10:in `visit'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/root_node.rb:36:in `css_tree'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/tree/root_node.rb:20:in `render'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-3.7.4/lib/sass/engine.rb:290:in `render'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sass-rails-5.0.7/lib/sass/rails/template.rb:47:in `evaluate'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/tilt-2.0.9/lib/tilt/template.rb:109:in `render'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy_tilt_processor.rb:25:in `call'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:75:in `call_processor'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `reverse_each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:134:in `load_from_unloaded'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:60:in `block in load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:44:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:20:in `block in initialize'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:47:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/bundle.rb:23:in `block in call'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/utils.rb:200:in `dfs'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/bundle.rb:24:in `call'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:75:in `call_processor'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `reverse_each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/processor_utils.rb:56:in `call_processors'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:134:in `load_from_unloaded'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:60:in `block in load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/loader.rb:44:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:20:in `block in initialize'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/cached_environment.rb:47:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/base.rb:66:in `find_asset'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/base.rb:73:in `find_all_linked_assets'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/manifest.rb:142:in `block in find'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:114:in `block (2 levels) in logical_paths'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:228:in `block in stat_tree'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:212:in `block in stat_directory'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:209:in `each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:209:in `stat_directory'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/path_utils.rb:227:in `stat_tree'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:105:in `each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:105:in `block in logical_paths'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:104:in `each'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/legacy.rb:104:in `logical_paths'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/manifest.rb:140:in `find'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/sprockets/manifest.rb:186:in `compile'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/task.rb:68:in `block (3 levels) in define'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-3.7.2/lib/rake/sprocketstask.rb:147:in `with_logger'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /home/gitlab-customers/customers-gitlab-com/vendor/bundle/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/task.rb:67:in `block (2 levels) in define'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /usr/local/bin/bundle:23:in `load'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: /usr/local/bin/bundle:23:in `<main>'
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: Tasks: TOP => assets:precompile

I wouldn't expect the cookbook to continue with the HUP after that, but it did:

Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: executable=/opt/chef/bin/chef-client
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: [2019-10-07T13:25:54+00:00] INFO: file[/home/gitlab-customers/customers-gitlab-com/tmp/VERSION] sending run action to execute[db:migrate] (delayed)
Oct 07 13:25:54 customers.gitlab.com chef-client[80484]: [2019-10-07T13:25:54+00:00] INFO: Processing execute[db:migrate] action run (cookbook-customers-gitlab-com::prd-setup line 82)
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] INFO: execute[db:migrate] ran successfully
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] INFO: execute[db:migrate] sending run action to execute[HUP unicorn] (delayed)
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] INFO: Processing execute[HUP unicorn] action run (cookbook-customers-gitlab-com::prd-setup line 91)
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] INFO: execute[HUP unicorn] ran successfully
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] ERROR: Running exception handlers
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] ERROR: Exception handlers complete
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] ERROR: execute[precompile:assets] (cookbook-customers-gitlab-com::prd-setup line 73) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: ---- Begin output of /usr/local/bin/bundle

Oct 07 13:26:03 customers.gitlab.com chef-client[80484]: [2019-10-07T13:26:03+00:00] INFO: execute[db:migrate] sending run action to execute[HUP unicorn] (delayed)

@jameslopez The db:migrate also notifies the HUP notification and it looks like that was successful.

right, but we should have stopped at precompile:assets

execute 'precompile:assets' do
  command '/usr/local/bin/bundle exec /usr/local/bin/rake assets:precompile RAILS_ENV=production'
  cwd '/home/gitlab-customers/customers-gitlab-com'
  user 'gitlab-customers'
  group 'gitlab-customers'
  action :nothing
  notifies :run, 'execute[HUP unicorn]'
end

so perhaps we should move the db:migrate to right after the assets. Would that stop the thing as they are not in parallel?

file '/home/gitlab-customers/customers-gitlab-com/tmp/VERSION' do
  owner 'gitlab-customers'
  group 'gitlab-customers'
  content `git -C /home/gitlab-customers/customers-gitlab-com rev-parse HEAD`
  notifies :run, 'execute[precompile:assets]', :delayed
  notifies :run, 'execute[db:migrate]', :delayed
end

reopened

Hi @mwasilewski-gitlab,

This issue is labelled incident or change and we would like some more information about it. Please set the severity appropriately as defined by

• change severities
• incident severities

Thanks for your help!

---

You are welcome to help improve this comment.

closed

@jameslopez we try to keep the incident issue open only for currently ongoing incidents, i.e. open incident issues indicate that we are still experiencing problems. Any investigation that is required as a follow up is better fitted in an RCA (Root Cause Analysis) issue

Sorry about that. We created https://gitlab.com/gitlab-org/fulfillment-meta/issues/53 as the RCA. Thanks!

This incident was closed before the IncidentReview-Completed label was applied. The issue is being reopened so that it will appear on the Production Incidents board—and can be moved through the entire incident management workflow.

Please review the Incident Workflow section on the Incident Management handbook page for more information.

/open

This incident was closed before the IncidentResolved label was applied applied—or after a review was started by not completed. The issue is being reopened so that it will appear on the Production Incidents board. Please mark it as either IncidentResolved or IncidentReview-Completed before closing.

Please review the Incident Workflow section on the Incident Management handbook page for more information.

/open

This incident was closed before the IncidentReview-Completed label was applied. The issue is being reopened so that it will appear on the Production Incidents board—and can be moved through the entire incident management workflow.

Please review the Incident Workflow section on the Incident Management handbook page for more information.

/open

added severity4 label

changed the severity to Low - S4