Labels API is slow

Trying to fetch data locally to analyze, I noticed that labels are very slow to fetch via /api/v4/projects/:id/labels: https://dashboards.gitlab.net/d/000000126/grape-endpoints?orgId=1&var-action=Grape%23GET%20%2Fapi%2Fprojects%2F:id%2Flabels&var-database=influxdb-01-inf-gprd

Data on the dashboards validates that:

This seems like an N + 1 of the counts for the following fields:

1. open_issues_count
2. closed_issues_count
3. open_merge_requests_count

/cc @stanhu @smcgivern