Group packages API is slow for large groups

The /api/v4/groups/:name/packages endpoint is slow for very large groups.

The slow method is:

    def group_projects_visible_to_current_user
      ::Project
        .in_namespace(groups)
        .public_or_visible_to_user(current_user, Gitlab::Access::REPORTER)
        .with_project_feature
        .select { |project| Ability.allowed?(current_user, :read_package, project) }
    end

https://gitlab.com/gitlab-org/gitlab/blob/147ad05bb19727443393762b585ada27c31a3b15/app/finders/packages/group_packages_finder.rb#L38-44

In this example, Ability.allowed? was called ~11k times.

gitlab-com/gl-infra/production#2815 (closed) has more details including a profile of a request to the URL.