Restore N+1 query fix reverted in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16118

In https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16118 the fix introduced in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15788 was reverted because it caused the problems described in https://gitlab.com/gitlab-org/gitlab-ce/issues/41327.

We need to restore this fix somehow without it re-introducing the same problem as the N+1 query problems are quite significant.