Gitaly Client: encapsulate GRPC errors

Problem to solve

Encapsulate all expected GRPC exceptions into GitlayClient own exceptions, and provide extra context and meaningful errors.

Motivation here is to provide Human Errors that provide "Context", "Feedback" and "Recovery": See talk: https://olivierlacan.com/talks/human-errors/

By encapsulating the GRPC errors, we now have something we can rescue on the application side instead of a generic 503 error page.

We can use this new exceptions to degrade gracefully (ex: render standard layout with all menus and headers and a nice message telling user that repository storage is under heavy load, to retry after waiting a few minutes.

Providing this type of information is better than a generic 503. Instead of "hey GitLab is buggy", they will understand "Hey, everybody is using GitLab so they are having scalling problems right now".

Further details

When gitaly_client.rb talks to remote gitaly-go via GRPC, there are many different errors that can happen. It should rescue the ones it is aware of and encapsulates so we can safely rescue and/or provide meaningful errors when needed.

One example:

GRPC::DeadlineExceeded. We expect this to happen, even if we prefer then not to. This happens almost everytime you start GDK with cold cache.

Instead of a big stacktrace without helpful information:

``` GRPC::DeadlineExceeded - 4:Deadline Exceeded: grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status' grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call' grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/active_call.rb:372:in `request_response' grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response' grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!' grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response' grpc-1.11.0-universal (darwin) src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class' lib/gitlab/gitaly_client.rb:134:in `call' lib/gitlab/gitaly_client/repository_service.rb:17:in `exists?' lib/gitlab/git/repository.rb:147:in `exists?' app/models/repository.rb:506:in `exists?' lib/gitlab/repository_cache_adapter.rb:17:in `block (2 levels) in cache_method' activesupport (4.2.10) lib/active_support/cache.rb:299:in `block in fetch' activesupport (4.2.10) lib/active_support/cache.rb:585:in `block in save_block_result_to_cache' activesupport (4.2.10) lib/active_support/cache.rb:547:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications.rb:166:in `instrument' activesupport (4.2.10) lib/active_support/cache.rb:547:in `instrument' activesupport (4.2.10) lib/active_support/cache.rb:584:in `save_block_result_to_cache' activesupport (4.2.10) lib/active_support/cache.rb:299:in `fetch' lib/gitlab/repository_cache.rb:22:in `fetch' lib/gitlab/repository_cache_adapter.rb:59:in `cache_method_output' lib/gitlab/repository_cache_adapter.rb:16:in `block in cache_method' app/models/repository.rb:110:in `commit' app/models/project.rb:549:in `commit' lib/gitlab/cache/ci/project_pipeline_status.rb:10:in `commit' lib/gitlab/cache/ci/project_pipeline_status.rb:83:in `load_from_project' lib/gitlab/cache/ci/project_pipeline_status.rb:75:in `load_status' lib/gitlab/cache/ci/project_pipeline_status.rb:21:in `block in load_in_batch_for_projects' lib/gitlab/cache/ci/project_pipeline_status.rb:19:in `load_in_batch_for_projects' app/helpers/projects_helper.rb:193:in `load_pipeline_status' app/views/shared/projects/_list.html.haml:14:in `_app_views_shared_projects__list_html_haml___823629925720167338_2416731300' actionview (4.2.10) lib/action_view/template.rb:145:in `block in render' activesupport (4.2.10) lib/active_support/notifications.rb:166:in `instrument' actionview (4.2.10) lib/action_view/template.rb:333:in `instrument' actionview (4.2.10) lib/action_view/template.rb:143:in `render' actionview (4.2.10) lib/action_view/renderer/partial_renderer.rb:339:in `render_partial' actionview (4.2.10) lib/action_view/renderer/partial_renderer.rb:310:in `block in render' actionview (4.2.10) lib/action_view/renderer/abstract_renderer.rb:39:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `instrument' actionview (4.2.10) lib/action_view/renderer/abstract_renderer.rb:39:in `instrument' actionview (4.2.10) lib/action_view/renderer/partial_renderer.rb:309:in `render' actionview (4.2.10) lib/action_view/renderer/renderer.rb:51:in `render_partial' actionview (4.2.10) lib/action_view/helpers/rendering_helper.rb:35:in `render' haml (4.0.7) lib/haml/helpers/action_view_mods.rb:12:in `render_with_haml' app/views/dashboard/projects/_projects.html.haml:1:in `_app_views_dashboard_projects__projects_html_haml__494939521040164000_2416874600' actionview (4.2.10) lib/action_view/template.rb:145:in `block in render' activesupport (4.2.10) lib/active_support/notifications.rb:166:in `instrument' actionview (4.2.10) lib/action_view/template.rb:333:in `instrument' actionview (4.2.10) lib/action_view/template.rb:143:in `render' actionview (4.2.10) lib/action_view/renderer/partial_renderer.rb:339:in `render_partial' actionview (4.2.10) lib/action_view/renderer/partial_renderer.rb:310:in `block in render' actionview (4.2.10) lib/action_view/renderer/abstract_renderer.rb:39:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `instrument' actionview (4.2.10) lib/action_view/renderer/abstract_renderer.rb:39:in `instrument' actionview (4.2.10) lib/action_view/renderer/partial_renderer.rb:309:in `render' actionview (4.2.10) lib/action_view/renderer/renderer.rb:51:in `render_partial' actionview (4.2.10) lib/action_view/helpers/rendering_helper.rb:35:in `render' haml (4.0.7) lib/haml/helpers/action_view_mods.rb:12:in `render_with_haml' app/views/dashboard/projects/index.html.haml:15:in `_app_views_dashboard_projects_index_html_haml___1480420427507514207_2450099100' actionview (4.2.10) lib/action_view/template.rb:145:in `block in render' activesupport (4.2.10) lib/active_support/notifications.rb:166:in `instrument' actionview (4.2.10) lib/action_view/template.rb:333:in `instrument' actionview (4.2.10) lib/action_view/template.rb:143:in `render' actionview (4.2.10) lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template' actionview (4.2.10) lib/action_view/renderer/abstract_renderer.rb:39:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `instrument' actionview (4.2.10) lib/action_view/renderer/abstract_renderer.rb:39:in `instrument' actionview (4.2.10) lib/action_view/renderer/template_renderer.rb:53:in `block in render_template' actionview (4.2.10) lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout' actionview (4.2.10) lib/action_view/renderer/template_renderer.rb:52:in `render_template' actionview (4.2.10) lib/action_view/renderer/template_renderer.rb:14:in `render' actionview (4.2.10) lib/action_view/renderer/renderer.rb:46:in `render_template' actionview (4.2.10) lib/action_view/renderer/renderer.rb:27:in `render' actionview (4.2.10) lib/action_view/rendering.rb:100:in `_render_template' actionpack (4.2.10) lib/action_controller/metal/streaming.rb:217:in `_render_template' actionview (4.2.10) lib/action_view/rendering.rb:83:in `render_to_body' actionpack (4.2.10) lib/action_controller/metal/rendering.rb:32:in `render_to_body' actionpack (4.2.10) lib/action_controller/metal/renderers.rb:37:in `render_to_body' actionpack (4.2.10) lib/abstract_controller/rendering.rb:25:in `render' actionpack (4.2.10) lib/action_controller/metal/rendering.rb:16:in `render' actionpack (4.2.10) lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render' activesupport (4.2.10) lib/active_support/core_ext/benchmark.rb:12:in `block in ms' /Users/brodock/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/benchmark.rb:308:in `realtime' activesupport (4.2.10) lib/active_support/core_ext/benchmark.rb:12:in `ms' actionpack (4.2.10) lib/action_controller/metal/instrumentation.rb:44:in `block in render' actionpack (4.2.10) lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime' activerecord (4.2.10) lib/active_record/railties/controller_runtime.rb:25:in `cleanup_view_runtime' elasticsearch-rails (0.1.9) lib/elasticsearch/rails/instrumentation/controller_runtime.rb:20:in `cleanup_view_runtime' actionpack (4.2.10) lib/action_controller/metal/instrumentation.rb:43:in `render' actionpack (4.2.10) lib/action_controller/metal/mime_responds.rb:217:in `respond_to' app/controllers/dashboard/projects_controller.rb:12:in `index' app/controllers/root_controller.rb:18:in `block in index' lib/gitlab/gitaly_client.rb:333:in `allow_n_plus_1_calls' app/controllers/root_controller.rb:17:in `index' actionpack (4.2.10) lib/action_controller/metal/implicit_render.rb:4:in `send_action' actionpack (4.2.10) lib/abstract_controller/base.rb:198:in `process_action' actionpack (4.2.10) lib/action_controller/metal/rendering.rb:10:in `process_action' actionpack (4.2.10) lib/abstract_controller/callbacks.rb:20:in `block in process_action' activesupport (4.2.10) lib/active_support/callbacks.rb:117:in `call' activesupport (4.2.10) lib/active_support/callbacks.rb:555:in `block (2 levels) in compile' activesupport (4.2.10) lib/active_support/callbacks.rb:505:in `call' activesupport (4.2.10) lib/active_support/callbacks.rb:498:in `block (2 levels) in around' activesupport (4.2.10) lib/active_support/callbacks.rb:313:in `block (2 levels) in halting' lib/gitlab/i18n.rb:51:in `with_locale' lib/gitlab/i18n.rb:57:in `with_user_locale' app/controllers/application_controller.rb:370:in `set_locale' activesupport (4.2.10) lib/active_support/callbacks.rb:432:in `block in make_lambda' activesupport (4.2.10) lib/active_support/callbacks.rb:312:in `block in halting' activesupport (4.2.10) lib/active_support/callbacks.rb:497:in `block in around' activesupport (4.2.10) lib/active_support/callbacks.rb:505:in `call' activesupport (4.2.10) lib/active_support/callbacks.rb:498:in `block (2 levels) in around' activesupport (4.2.10) lib/active_support/callbacks.rb:298:in `block in halting_and_conditional' activesupport (4.2.10) lib/active_support/callbacks.rb:497:in `block in around' activesupport (4.2.10) lib/active_support/callbacks.rb:505:in `call' activesupport (4.2.10) lib/active_support/callbacks.rb:498:in `block (2 levels) in around' activesupport (4.2.10) lib/active_support/callbacks.rb:313:in `block (2 levels) in halting' sentry-raven (2.7.2) lib/raven/integrations/rails/controller_transaction.rb:7:in `block in included' activesupport (4.2.10) lib/active_support/callbacks.rb:441:in `block in make_lambda' activesupport (4.2.10) lib/active_support/callbacks.rb:312:in `block in halting' activesupport (4.2.10) lib/active_support/callbacks.rb:497:in `block in around' activesupport (4.2.10) lib/active_support/callbacks.rb:505:in `call' activesupport (4.2.10) lib/active_support/callbacks.rb:92:in `__run_callbacks__' activesupport (4.2.10) lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks' activesupport (4.2.10) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (4.2.10) lib/abstract_controller/callbacks.rb:19:in `process_action' actionpack (4.2.10) lib/action_controller/metal/rescue.rb:29:in `process_action' actionpack (4.2.10) lib/action_controller/metal/instrumentation.rb:32:in `block in process_action' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `block in instrument' activesupport (4.2.10) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (4.2.10) lib/active_support/notifications.rb:164:in `instrument' actionpack (4.2.10) lib/action_controller/metal/instrumentation.rb:30:in `process_action' actionpack (4.2.10) lib/action_controller/metal/params_wrapper.rb:250:in `process_action' activerecord (4.2.10) lib/active_record/railties/controller_runtime.rb:18:in `process_action' actionpack (4.2.10) lib/abstract_controller/base.rb:137:in `process' actionview (4.2.10) lib/action_view/rendering.rb:30:in `process' actionpack (4.2.10) lib/action_controller/metal.rb:196:in `dispatch' actionpack (4.2.10) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch' actionpack (4.2.10) lib/action_controller/metal.rb:237:in `block in action' actionpack (4.2.10) lib/action_dispatch/routing/route_set.rb:74:in `dispatch' actionpack (4.2.10) lib/action_dispatch/routing/route_set.rb:43:in `serve' actionpack (4.2.10) lib/action_dispatch/journey/router.rb:43:in `block in serve' actionpack (4.2.10) lib/action_dispatch/journey/router.rb:30:in `serve' actionpack (4.2.10) lib/action_dispatch/routing/route_set.rb:817:in `call' lib/gitlab/middleware/multipart.rb:97:in `call' lib/gitlab/request_profiler/middleware.rb:14:in `call' lib/gitlab/query_limiting/middleware.rb:17:in `block in call' lib/gitlab/query_limiting/transaction.rb:37:in `run' lib/gitlab/query_limiting/middleware.rb:16:in `call' ee/lib/gitlab/jira/middleware.rb:15:in `call' lib/gitlab/middleware/go.rb:17:in `call' lib/gitlab/etag_caching/middleware.rb:11:in `call' batch-loader (1.2.1) lib/batch_loader/middleware.rb:11:in `call' rack-attack (4.4.1) lib/rack/attack.rb:107:in `call' warden (1.2.7) lib/warden/manager.rb:36:in `block in call' warden (1.2.7) lib/warden/manager.rb:35:in `call' rack-cors (1.0.2) lib/rack/cors.rb:97:in `call' rack (1.6.10) lib/rack/etag.rb:24:in `call' rack (1.6.10) lib/rack/conditionalget.rb:25:in `call' rack (1.6.10) lib/rack/head.rb:13:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/params_parser.rb:27:in `call' lib/gitlab/middleware/read_only/controller.rb:40:in `call' lib/gitlab/middleware/read_only.rb:16:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/flash.rb:260:in `call' rack (1.6.10) lib/rack/session/abstract/id.rb:225:in `context' rack (1.6.10) lib/rack/session/abstract/id.rb:220:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/cookies.rb:560:in `call' activerecord (4.2.10) lib/active_record/query_cache.rb:36:in `call' activerecord (4.2.10) lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call' activerecord (4.2.10) lib/active_record/migration.rb:377:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call' activesupport (4.2.10) lib/active_support/callbacks.rb:88:in `__run_callbacks__' activesupport (4.2.10) lib/active_support/callbacks.rb:778:in `_run_call_callbacks' activesupport (4.2.10) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (4.2.10) lib/action_dispatch/middleware/callbacks.rb:27:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/reloader.rb:73:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/remote_ip.rb:78:in `call' better_errors (2.1.1) lib/better_errors/middleware.rb:84:in `protected_app_call' better_errors (2.1.1) lib/better_errors/middleware.rb:79:in `better_errors_call' better_errors (2.1.1) lib/better_errors/middleware.rb:57:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call' railties (4.2.10) lib/rails/rack/logger.rb:38:in `call_app' railties (4.2.10) lib/rails/rack/logger.rb:20:in `block in call' activesupport (4.2.10) lib/active_support/tagged_logging.rb:68:in `block in tagged' activesupport (4.2.10) lib/active_support/tagged_logging.rb:26:in `tagged' activesupport (4.2.10) lib/active_support/tagged_logging.rb:68:in `tagged' railties (4.2.10) lib/rails/rack/logger.rb:20:in `call' sprockets-rails (3.2.1) lib/sprockets/rails/quiet_assets.rb:13:in `call' lib/gitlab/request_context.rb:18:in `call' request_store (1.3.1) lib/request_store/middleware.rb:9:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/request_id.rb:21:in `call' rack (1.6.10) lib/rack/methodoverride.rb:22:in `call' rack (1.6.10) lib/rack/runtime.rb:18:in `call' rack (1.6.10) lib/rack/lock.rb:17:in `call' actionpack (4.2.10) lib/action_dispatch/middleware/static.rb:120:in `call' lib/gitlab/middleware/static.rb:9:in `call' lib/gitlab/webpack/dev_server_middleware.rb:25:in `perform_request' rack-proxy (0.6.0) lib/rack/proxy.rb:57:in `call' rack (1.6.10) lib/rack/sendfile.rb:113:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call' sentry-raven (2.7.2) lib/raven/integrations/rack.rb:51:in `call' railties (4.2.10) lib/rails/engine.rb:518:in `call' railties (4.2.10) lib/rails/application.rb:165:in `call' railties (4.2.10) lib/rails/railtie.rb:194:in `method_missing' lib/gitlab/middleware/release_env.rb:10:in `call' rack (1.6.10) lib/rack/urlmap.rb:66:in `block in call' rack (1.6.10) lib/rack/urlmap.rb:50:in `call' thin (1.7.0) lib/thin/connection.rb:86:in `block in pre_process' thin (1.7.0) lib/thin/connection.rb:84:in `pre_process' thin (1.7.0) lib/thin/connection.rb:53:in `process' thin (1.7.0) lib/thin/connection.rb:39:in `receive_data' eventmachine (1.0.8) lib/eventmachine.rb:193:in `run' thin (1.7.0) lib/thin/backends/base.rb:73:in `start' thin (1.7.0) lib/thin/server.rb:162:in `start' thin (1.7.0) lib/thin/controllers/controller.rb:87:in `start' thin (1.7.0) lib/thin/runner.rb:200:in `run_command' thin (1.7.0) lib/thin/runner.rb:156:in `run!' thin (1.7.0) bin/thin:6:in `' () Users/brodock/.rvm/gems/ruby-2.4.4/bin/thin:23:in `' ```

Proposal

Identify all common GRPC exceptions that we expect to happen and encapsulate from gitaly_client and enhance with context metadata.

(see example below)

What does success look like, and how can we measure that?

As an exemple of how we can improve the Exceptions:

Instead of previous GRPC::DeadlineExceeded would be nice to have something like:

Gitaly::TimeoutError - "Gitaly client for storage: '#{ex.storage}' tried to communicate with service: `#{ex.service}`, rpc: `#{ex.rpc}` but it took more than timeout limit: `#{ex.timeout}` to receive a response"
lib/gitlab/gitaly_client.rb:134:in `call'
  lib/gitlab/gitaly_client/repository_service.rb:17:in `exists?'
  lib/gitlab/git/repository.rb:147:in `exists?'
  app/models/repository.rb:506:in `exists?'
...

Links / references

https://olivierlacan.com/talks/human-errors/