Next iteration of `pipelines.json` performance improvements

Problem

We've shipped a few iterations of pipelines.json endpoint performance improvement so far (the endpoint is used to show a pipelines list page).

A few recent improvements:

• !34160 (merged)
• https://gitlab.com/gitlab-org/gitlab-ee/issues/9549

The endpoint is still slow and we need to investigate and profile it again.