Build commit graph with Bloom Filter
In our monorepo, we started to observe very slow response time in FindCommits
grpc.
User when edit Merge Requests would get timeout with the following stack trace:
Gitlab::Git::CommandError (4:Deadline Exceeded):
lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
lib/gitlab/git/repository.rb:347:in `log'
lib/gitlab/git/commit.rb:46:in `where'
app/models/repository.rb:155:in `commits'
lib/gitlab/metrics/instrumentation.rb:161:in `block in commits'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `commits'
ee/lib/gitlab/authority_analyzer.rb:26:in `involved_users'
ee/lib/gitlab/authority_analyzer.rb:15:in `calculate'
ee/app/presenters/merge_request_approver_presenter.rb:56:in `users_from_git_log_authors'
ee/app/presenters/merge_request_approver_presenter.rb:44:in `block in users'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
ee/app/presenters/merge_request_approver_presenter.rb:43:in `users'
ee/app/presenters/merge_request_approver_presenter.rb:25:in `any?'
ee/app/views/shared/issuable/_approvals.html.haml:25
app/helpers/application_helper.rb:10:in `render_if_exists'
app/views/shared/issuable/_form.html.haml:36
app/views/projects/merge_requests/_form.html.haml:3
app/views/projects/merge_requests/_form.html.haml:1
app/views/projects/merge_requests/edit.html.haml:5
app/controllers/application_controller.rb:133:in `render'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
app/controllers/application_controller.rb:497:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:488:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:482:in `set_locale'
lib/gitlab/error_tracking.rb:51:in `with_context'
app/controllers/application_controller.rb:547:in `sentry_context'
app/controllers/application_controller.rb:475:in `block in set_current_context'
lib/gitlab/application_context.rb:52:in `block in use'
lib/gitlab/application_context.rb:52:in `use'
lib/gitlab/application_context.rb:20:in `with_context'
app/controllers/application_controller.rb:468:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:24:in `call'
lib/gitlab/metrics/redis_rack_middleware.rb:22:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:54:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
ee/lib/gitlab/database/load_balancing/rack_middleware.rb:39:in `call'
ee/lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:125:in `call'
lib/gitlab/middleware/read_only/controller.rb:51:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
What this is suggesting is that https://gitlab.com/gitlab-org/gitlab/-/blob/v13.3.8-ee/ee/lib/gitlab/authority_analyzer.rb#L26 is being called and get timeout by Gitaly.
I suspect this is because in a big monorepo, git log -- <file_path>
could be very slow as reasons indicated in https://devblogs.microsoft.com/devops/super-charging-the-git-commit-graph-iv-bloom-filters/
The solution here is to build the commit-graph with Bloom filter included by adding --changed-paths
to git commit-graph write
. I have a draft solution in sluongng/gitaly@78dba8b7 that need to be developed into a full fledged solution.