Commits in fork can have impact on original project (again)

We already fixed this issue few months ago. It's weird that it's introduced again.

It looks like @dblessing updated his fork and that "regular" action closed issue https://gitlab.com/gitlab-org/gitlab-ee/issues/835

I think it's related to recent work on performance improvements for references.

Also we should cover this case with tests to not introduce this bug third time