Metrics dashboard validation regressions 500 errors
Summary
!38925 (merged) introduced regression on staging.gitlab.com which causes getting dashboard validation information to fail with 500 error.
Steps to reproduce
Visit any YAML file at .gitalb/dashboards directory in any projects repository.
Or visit metrics section in any project with metrics section enabled
Example Project
Should replicate for every project on staging.
What is the current bug behavior?
Trying to obtain metrics dashboard information validation fails with 500 error code
What is the expected correct behavior?
Metrics dashboard validation information can be fetched.
Relevant logs and/or screenshots
Errno::ENOENT: No such file or directory @ realpath_rec - /var/opt/gitlab/gitlab-rails/working/lib
from json_schemer.rb:52:in `realpath'
from json_schemer.rb:52:in `realpath'
from json_schemer.rb:52:in `schema'
from gitlab/metrics/dashboard/validator/client.rb:44:in `schemer'
from gitlab/metrics/dashboard/validator/client.rb:48:in `validate_against_schema'
from gitlab/metrics/dashboard/validator/client.rb:21:in `execute'
from gitlab/metrics/dashboard/validator.rb:22:in `errors'
from blob_viewer/metrics_dashboard_yml.rb:30:in `parse_blob_data'
from blob_viewer/metrics_dashboard_yml.rb:21:in `block in errors'
from gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
from blob_viewer/metrics_dashboard_yml.rb:19:in `errors'
from blob_viewer/metrics_dashboard_yml.rb:15:in `valid?'
from /opt/gitlab/embedded/service/gitlab-rails/app/views/projects/blob/viewers/_metrics_dashboard_yml.html.haml:1:in `_app_views_projects_blob_viewers__metrics_dashboard_yml_html_haml__3280094965249438028_69981267223360'
from action_view/base.rb:274:in `_run'
from action_view/template.rb:185:in `block in render'
from active_support/notifications.rb:182:in `instrument'
from action_view/template.rb:385:in `instrument_render_template'
from action_view/template.rb:183:in `render'
from action_view/renderer/partial_renderer.rb:357:in `block in render_partial'
from action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
from active_support/notifications.rb:182:in `instrument'
from action_view/renderer/abstract_renderer.rb:87:in `instrument'
from action_view/renderer/partial_renderer.rb:346:in `render_partial'
from action_view/renderer/partial_renderer.rb:317:in `render'
from action_view/renderer/renderer.rb:65:in `render_partial_to_object'
from action_view/renderer/renderer.rb:53:in `render_partial'
from action_view/helpers/rendering_helper.rb:38:in `render'
from /opt/gitlab/embedded/service/gitlab-rails/app/views/projects/blob/_viewer.html.haml:23:in `_app_views_projects_blob__viewer_html_haml__116990261264358598_69981248657000'
from action_view/base.rb:274:in `_run'
from action_view/template.rb:185:in `block in render'
from active_support/notifications.rb:182:in `instrument'
from action_view/template.rb:385:in `instrument_render_template'
from action_view/template.rb:183:in `render'
from action_view/renderer/template_renderer.rb:58:in `block (2 levels) in render_template'
from action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
from active_support/notifications.rb:180:in `block in instrument'
from active_support/notifications/instrumenter.rb:24:in `instrument'
from active_support/notifications.rb:180:in `instrument'
from action_view/renderer/abstract_renderer.rb:87:in `instrument'
from action_view/renderer/template_renderer.rb:57:in `block in render_template'
from action_view/renderer/template_renderer.rb:65:in `render_with_layout'
from action_view/renderer/template_renderer.rb:56:in `render_template'
from action_view/renderer/template_renderer.rb:13:in `render'
from action_view/renderer/renderer.rb:61:in `render_template_to_object'
from action_view/renderer/renderer.rb:29:in `render_to_object'
from action_view/rendering.rb:117:in `block in _render_template'
from action_view/base.rb:304:in `in_rendering_context'
from action_view/rendering.rb:116:in `_render_template'
from action_controller/metal/streaming.rb:218:in `_render_template'
from action_view/rendering.rb:103:in `render_to_body'
from action_controller/metal/rendering.rb:52:in `render_to_body'
from action_controller/metal/renderers.rb:142:in `render_to_body'
from abstract_controller/rendering.rb:46:in `render_to_string'
from action_controller/metal/rendering.rb:41:in `render_to_string'
from application_controller.rb:349:in `view_to_html_string'
from renders_blob.rb:20:in `blob_json'
from projects/blob_controller.rb:247:in `show_json'
from projects/blob_controller.rb:60:in `block (2 levels) in show'
from action_controller/metal/mime_responds.rb:214:in `respond_to'
from projects/blob_controller.rb:52:in `show'
from action_controller/metal/basic_implicit_render.rb:6:in `send_action'
from abstract_controller/base.rb:195:in `process_action'
from action_controller/metal/rendering.rb:30:in `process_action'
from abstract_controller/callbacks.rb:42:in `block in process_action'
from active_support/callbacks.rb:112:in `block in run_callbacks'
from application_controller.rb:546:in `block in allow_gitaly_ref_name_caching'
from gitlab/gitaly_client.rb:318:in `allow_ref_name_caching'
from application_controller.rb:545:in `allow_gitaly_ref_name_caching'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from gitlab/ip_address_state.rb:10:in `with'
from ee/application_controller.rb:44:in `set_current_ip_address'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from application_controller.rb:491:in `set_current_admin'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from gitlab/session.rb:11:in `with_session'
from application_controller.rb:482:in `set_session_storage'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from gitlab/i18n.rb:73:in `with_locale'
from gitlab/i18n.rb:79:in `with_user_locale'
from application_controller.rb:476:in `set_locale'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from gitlab/error_tracking.rb:50:in `with_context'
from application_controller.rb:541:in `sentry_context'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from raven/integrations/rails/controller_transaction.rb:7:in `block in included'
from active_support/callbacks.rb:121:in `instance_exec'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from marginalia.rb:111:in `record_query_comment'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from application_controller.rb:469:in `block in set_current_context'
from gitlab/application_context.rb:52:in `block in use'
from labkit/context.rb:32:in `with_context'
from gitlab/application_context.rb:52:in `use'
from gitlab/application_context.rb:20:in `with_context'
from application_controller.rb:462:in `set_current_context'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from active_support/callbacks.rb:139:in `run_callbacks'
from abstract_controller/callbacks.rb:41:in `process_action'
from action_controller/metal/rescue.rb:22:in `process_action'
from action_controller/metal/instrumentation.rb:33:in `block in process_action'
from active_support/notifications.rb:180:in `block in instrument'
from active_support/notifications/instrumenter.rb:24:in `instrument'
from active_support/notifications.rb:180:in `instrument'
from action_controller/metal/instrumentation.rb:32:in `process_action'
from action_controller/metal/params_wrapper.rb:245:in `process_action'
from active_record/railties/controller_runtime.rb:27:in `process_action'
from abstract_controller/base.rb:136:in `process'
from action_view/rendering.rb:39:in `process'
from action_controller/metal.rb:190:in `dispatch'
from action_controller/metal.rb:254:in `dispatch'
from action_dispatch/routing/route_set.rb:50:in `dispatch'
from action_dispatch/routing/route_set.rb:33:in `serve'
from action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
from action_dispatch/routing/mapper.rb:48:in `serve'
from action_dispatch/journey/router.rb:49:in `block in serve'
from action_dispatch/journey/router.rb:32:in `each'
from action_dispatch/journey/router.rb:32:in `serve'
from action_dispatch/routing/route_set.rb:834:in `call'
from omniauth/strategy.rb:192:in `call!'
from omniauth/strategy.rb:169:in `call'
from omniauth/strategy.rb:420:in `call_app!'
from omni_auth/strategies/group_saml.rb:41:in `other_phase'
from omniauth/strategy.rb:190:in `call!'
from omniauth/strategy.rb:169:in `call'
from omniauth/strategy.rb:192:in `call!'
from omniauth/strategy.rb:169:in `call'
from omniauth/strategy.rb:192:in `call!'
from omniauth/strategy.rb:169:in `call'
from omniauth/strategy.rb:192:in `call!'
from omniauth/strategy.rb:169:in `call'
from omniauth/strategy.rb:192:in `call!'
from omniauth/strategy.rb:169:in `call'
from gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
from gitlab/middleware/rails_queue_duration.rb:33:in `call'
from gitlab/metrics/rack_middleware.rb:16:in `block in call'
from gitlab/metrics/transaction.rb:61:in `run'
from gitlab/metrics/rack_middleware.rb:16:in `call'
from gitlab/request_profiler/middleware.rb:17:in `call'
from gitlab/database/load_balancing/rack_middleware.rb:39:in `call'
from labkit/middleware/rack.rb:19:in `block in call'
from labkit/context.rb:32:in `with_context'
from labkit/middleware/rack.rb:18:in `call'
from gitlab/jira/middleware.rb:19:in `call'
from gitlab/middleware/go.rb:20:in `call'
from gitlab/etag_caching/middleware.rb:13:in `call'
from batch_loader/middleware.rb:11:in `call'
from rack/attack.rb:97:in `call'
from apollo_upload_server/middleware.rb:20:in `call'
from gitlab/middleware/multipart.rb:140:in `call'
from rack/attack.rb:111:in `call'
from warden/manager.rb:36:in `block in call'
from warden/manager.rb:34:in `catch'
from warden/manager.rb:34:in `call'
from rack/cors.rb:98:in `call'
from rack/tempfile_reaper.rb:15:in `call'
from rack/etag.rb:25:in `call'
from rack/conditional_get.rb:25:in `call'
from rack/head.rb:12:in `call'
from action_dispatch/http/content_security_policy.rb:18:in `call'
from gitlab/middleware/read_only/controller.rb:51:in `call'
from gitlab/middleware/read_only.rb:18:in `call'
from rack/session/abstract/id.rb:259:in `context'
from rack/session/abstract/id.rb:253:in `call'
from action_dispatch/middleware/cookies.rb:648:in `call'
from gitlab/middleware/same_site_cookies.rb:27:in `call'
from action_dispatch/middleware/callbacks.rb:27:in `block in call'
from active_support/callbacks.rb:101:in `run_callbacks'
from action_dispatch/middleware/callbacks.rb:26:in `call'
from action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
from action_dispatch/middleware/debug_exceptions.rb:32:in `call'
from action_dispatch/middleware/show_exceptions.rb:33:in `call'
from gitlab/middleware/basic_health_check.rb:25:in `call'
from lograge/rails_ext/rack/logger.rb:15:in `call_app'
from rails/rack/logger.rb:26:in `block in call'
from active_support/tagged_logging.rb:80:in `block in tagged'
from active_support/tagged_logging.rb:28:in `tagged'
from active_support/tagged_logging.rb:80:in `tagged'
from rails/rack/logger.rb:26:in `call'
from action_dispatch/middleware/remote_ip.rb:81:in `call'
from gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
from gitlab/middleware/request_context.rb:23:in `call'
from request_store/middleware.rb:19:in `call'
from action_dispatch/middleware/request_id.rb:27:in `call'
from rack/method_override.rb:22:in `call'
from rack/runtime.rb:22:in `call'
from rack/timeout/core.rb:123:in `block in call'
from rack/timeout/support/timeout.rb:19:in `timeout'
from rack/timeout/core.rb:122:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:9:in `call'
from action_dispatch/middleware/executor.rb:14:in `call'
from rack/sendfile.rb:111:in `call'
from action_dispatch/middleware/host_authorization.rb:76:in `call'
from gitlab/metrics/requests_rack_middleware.rb:60:in `call'
from raven/integrations/rack.rb:51:in `call'
from rails/engine.rb:527:in `call'
from rails/railtie.rb:190:in `public_send'
from rails/railtie.rb:190:in `method_missing'
from gitlab/middleware/release_env.rb:12:in `call'
from rack/urlmap.rb:68:in `block in call'
from rack/urlmap.rb:53:in `each'
from rack/urlmap.rb:53:in `call'
from puma/configuration.rb:228:in `call'
from puma/server.rb:691:in `handle_request'
from puma/server.rb:481:in `process_client'
from puma/server.rb:335:in `block in run'
from puma/thread_pool.rb:138:in `block in spawn_thread'Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by Mikołaj Wawrzyniak