Project.visible_to_user is rather slow in certain cases

Consider the following graph from the Method Calls dashboard:

Checking if a user has access to a project (likely something that runs on every project page) shouldn't take more than lets say 50 ms; definitely not between 130 and 500 ms.