Slow update of merge requests for forked project

I noticed that when I pushed a commit to a forked repo of gitlab-org/gitlab-ce in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32178, it took several minutes for the push to show up. This happened twice today to me.

The Kibana logs shows that FetchSourceBranch RPC took close to 4 minutes to complete:

 "json": {
      "correlation_id": "k7KO3Cmrpd3",
      "grpc.code": "OK",
      "grpc.start_time": "2019-08-24T21:15:40Z",
      "hostname": "file-08-stor-gprd",
      "grpc.request.topLevelGroup": "@hashed",
      "level": "info",
      "environment": "gprd",
      "peer.address": "10.220.6.27:37058",
      "grpc.request.repoStorage": "nfs-file08",
      "span.kind": "server",
      "grpc.method": "FetchSourceBranch",
      "msg": "finished unary call with code OK",
      "grpc.time_ms": 218661.11,
      "system": "grpc",
      "tag": "gitaly",
      "grpc.meta.client_name": "gitlab-sidekiq",
      "grpc.service": "gitaly.RepositoryService",
      "grpc.request.glProjectPath": "gitlab-org/gitlab-ce",
      "grpc.request.repoPath": "@hashed/94/d1/94d17a47b559db4e59e13ca10c0f1037ac49f3050e53adbf59ebc2feb528c344.git",
      "fqdn": "file-08-stor-gprd.c.gitlab-production.internal",
      "grpc.meta.auth_version": "v2",
      "grpc.request.glRepository": "project-13083",
      "grpc.request.fullMethod": "/gitaly.RepositoryService/FetchSourceBranch",
      "time": "2019-08-24T21:19:19Z"
    },

Note that the forked repo is in the same pool, and the source branch doesn't deviate from master all that much. Why does it take 3 minutes to fetch this branch to do the update?

@johncai @jacobvosmaer-gitlab For a project in the object pool, should we change UpdateMergeRequestsWorker to skip the FetchSourceBranch and just do the comparison directly? As shown below, I can do the diff directly without creating a temporary ref:

root@file-08-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/git-data/repositories/@hashed/81/af/81af24ceee70dce1a60fadf5971c7b7cc0782fd3c532fe4b2b87dbf0c9258cba.git# git diff patch-74..master
diff --git a/app/views/profiles/preferences/show.html.haml b/app/views/profiles/preferences/show.html.haml
index 0887e8e..d99063e 100644
--- a/app/views/profiles/preferences/show.html.haml
+++ b/app/views/profiles/preferences/show.html.haml
@@ -47,7 +47,7 @@
         = s_('Preferences|Layout width')
       = f.select :layout, layout_choices, {}, class: 'select2'
       .form-text.text-muted
-        = s_('Preferences|Choose between fixed (max. 1280px) and fluid (%{percentage}) application layout.').html_safe % { percentage: '100%' }
+        = s_('Preferences|Choose between fixed (max. 1280px) and fluid (100%%) application layout.')
     .form-group
       = f.label :dashboard, class: 'label-bold' do
         = s_('Preferences|Default dashboard')
diff --git a/locale/gitlab.pot b/locale/gitlab.pot
index 7dafb38..829b47d 100644
--- a/locale/gitlab.pot
+++ b/locale/gitlab.pot
@@ -8299,7 +8299,7 @@ msgstr ""
 msgid "Preferences|Behavior"
 msgstr ""

-msgid "Preferences|Choose between fixed (max. 1280px) and fluid (%{percentage}) application layout."
+msgid "Preferences|Choose between fixed (max. 1280px) and fluid (100%) application layout."
 msgstr ""

 msgid "Preferences|Choose what content you want to see on a project’s overview page."