We need to improve how Global Search handles corrupt or missing repositories
Blocked: this is blocked by the need to add reporting and better understand a systemic issue with index integrity.
Summary
We need to improve how Global Search Handles corrupt or missing repositories.
-
Determine a reproducible way to create the "not git repository" error -
Review how we can identify these scenarios -
Propose methods for better handling -
Collect additional input to select the approach that we want. -
TBD
If a repo path doesn't exist anymore for some reason,
"app/helpers/tree_helper.rb:18:in `tree_join'"
Experiences a no implicit conversion of nil into String
error which prevents any search result from being displayed.
Steps to reproduce
It's unclear how to get a project into that state (similarly in #334562 (closed) & #332195 (closed)).
But afterward: Use advanced search in the parent namespace with a term that works when searched only within the project.
Example Project
A customer reported this internally. All logs can be found there. Essentials below.
What is the current bug behavior?
What is the expected correct behavior?
Global search can ignore/skip such errors and surface those errors as warnings in the search results UI so that the search flow isn't completely disrupted and results from other projects are shown.
Relevant logs and/or screenshots
The SearchController#show error 500 is correlated with a grpc FindDefaultBranchName error
{ "severity": "DEBUG", "time": "…", "correlation_id": "…some…id…", "message": "Instantiating Gitlab::Auth::Ldap::Person with LDIF: …" }{ "method": "GET", "path": "/search", "format": "html", "controller": "SearchController", "action": "show", "status": 500, "time": "…", "params": [ { "key": "group_id", "value": "" }, { "key": "project_id", "value": "" }, { "key": "repository_ref", "value": "" }, { "key": "scope", "value": "blobs" }, { "key": "search", "value": "[FILTERED]" }, { "key": "snippets", "value": "false" } ], … "meta.caller_id": "SearchController#show", "meta.remote_ip": "10.201.7.111", "meta.feature_category": "global_search", "meta.client_id": "user/…", "meta.search.group_id": "", "meta.search.project_id": "", "meta.search.scope": "blobs", "meta.search.filters.confidential": null, "meta.search.filters.state": null, "meta.search.force_search_results": null, "gitaly_calls": 2, "gitaly_duration_s": 0.006053, "redis_calls": 9, "redis_duration_s": 0.002055, "redis_read_bytes": 1061, "redis_write_bytes": 1228, "redis_cache_calls": 6, "redis_cache_duration_s": 0.001257, "redis_cache_read_bytes": 724, "redis_cache_write_bytes": 359, "redis_shared_state_calls": 3, "redis_shared_state_duration_s": 0.000798, "redis_shared_state_read_bytes": 337, "redis_shared_state_write_bytes": 869, "elasticsearch_calls": 1, "elasticsearch_duration_s": 0.035125, "elasticsearch_timed_out_count": 0, "db_count": 28, "db_write_count": 5, "db_cached_count": 1, "external_http_count": 1, … "exception.class": "ActionView::Template::Error", "exception.message": "no implicit conversion of nil into String", "exception.backtrace": [ "app/helpers/tree_helper.rb:18:in `join'", "app/helpers/tree_helper.rb:18:in `tree_join'", "app/views/search/results/_blob.html.haml:3", "app/helpers/application_helper.rb:15:in `render_if_exists'", "app/views/search/_results.html.haml:27", "app/views/search/show.html.haml:22", "app/controllers/application_controller.rb:123:in `render'", "app/controllers/application_controller.rb:536:in `block in allow_gitaly_ref_name_caching'", "lib/gitlab/gitaly_client.rb:341:in `allow_ref_name_caching'", "app/controllers/application_controller.rb:535:in `allow_gitaly_ref_name_caching'", "ee/lib/gitlab/ip_address_state.rb:10:in `with'", "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'", "app/controllers/application_controller.rb:485:in `set_current_admin'", "lib/gitlab/session.rb:11:in `with_session'", "app/controllers/application_controller.rb:476:in `set_session_storage'", "lib/gitlab/i18n.rb:73:in `with_locale'", "lib/gitlab/i18n.rb:79:in `with_user_locale'", "app/controllers/application_controller.rb:470:in `set_locale'", "app/controllers/application_controller.rb:463:in `block in set_current_context'", "lib/gitlab/application_context.rb:70:in `block in use'", "lib/gitlab/application_context.rb:70:in `use'", "lib/gitlab/application_context.rb:27:in `with_context'", "app/controllers/application_controller.rb:454:in `set_current_context'", "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'", "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'", "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'", "lib/gitlab/metrics/transaction.rb:56:in `run'", "lib/gitlab/metrics/rack_middleware.rb:16:in `call'", "lib/gitlab/request_profiler/middleware.rb:17:in `call'", "lib/gitlab/jira/middleware.rb:19:in `call'", "lib/gitlab/middleware/go.rb:20:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/middleware/multipart.rb:172:in `call'", "lib/gitlab/middleware/read_only/controller.rb:50:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:21:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'", "lib/gitlab/middleware/release_env.rb:12:in `call'" ], … }{ "content_type": "text/html; charset=utf-8", "correlation_id": "…same…", "duration_ms": …, "host": "…", "level": "info", "method": "GET", "msg": "access", "proto": "HTTP/1.1", "referrer": "https://jira.…/", "remote_addr": "127.0.0.1:0", "remote_ip": "127.0.0.1", "route": "", "status": 500, … "uri": "/search?group_id=&project_id=&repository_ref=&scope=blobs&search=%220123-somestring%22+extension%3Ago&snippets=false", … } { "correlation_id": "…same…", "error": "rpc error: code = NotFound desc = GetRepoPath: not a git repository: \"/var/opt/gitlab/git-data/repositories/backend/legacy/common/event-mail-sender-old.git\"", "grpc.code": "NotFound", "grpc.meta.auth_version": "v2", "grpc.meta.client_name": "gitlab-web", "grpc.meta.deadline_type": "regular", "grpc.method": "FindDefaultBranchName", … } { "correlation_id": "…same…", "grpc.code": "OK", "grpc.meta.auth_version": "v2", "grpc.meta.client_name": "gitlab-web", "grpc.meta.deadline_type": "regular", "grpc.method": "RepositoryExists", "grpc.request.deadline": "…", "grpc.request.fullMethod": "/gitaly.RepositoryService/RepositoryExists", "grpc.request.glProjectPath": "…legacy/path…", "grpc.request.glRepository": "project-…", "grpc.request.repoPath": "…legacy/path…git", "grpc.request.repoStorage": "default", "grpc.request.topLevelGroup": "backend", "grpc.service": "gitaly.RepositoryService", "grpc.start_time": "…", "grpc.time_ms": 0.054, "level": "info", "msg": "finished unary call with code OK", "peer.address": "@", "pid": 648, "remote_ip": "10.201.7.111", "span.kind": "server", … }
Output of checks
GitLab v13.11.3. More details are requested from the customer.
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package, run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source, run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package, run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source, run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)