Revert Project.public_or_visible_to_user changes

What does this MR do?

These changes were introduced in MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17088. In https://gitlab.com/gitlab-com/infrastructure/issues/3772 we discovered these changes lead to a pretty drastic increase in SQL response timings.

We'll revert these changes so we can work on a better solution in the mean time without GitLab.com (or other installations) experiecing reduced performance in the mean time.

Impact of these changes:

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Review
  • Has been reviewed by Backend
  • Has been reviewed by Database
• Conform by the merge request performance guides
• Conform by the style guides
• Squashed related commits together
• End-to-end tests pass (package-qa manual pipeline job)

What are the relevant issue numbers?

https://gitlab.com/gitlab-com/infrastructure/issues/3772