Optimize the most resource-consuming SQL queries

The approach: analyze pg_stat_statements, order by total_time, go from top to bottom and optimize Top-N queries.

Reports K001, K002, K003 generated by postgres-checkup tool help to see the picture.

It's worth to start this process from the master node first and then proceeding to replicas. The final goal is decreasing resource consumption and, as a result, prepare better for the future growth.

Fresh checkups:

• 1h45m at ~2pm UTC 2019-04-09 (peak time)
• ~12min at ~8am UTC 2019-04-17

Issues opened:

MASTER:

• 50% of all load on master, very frequent check https://gitlab.com/gitlab-org/gitlab-ce/issues/60524 (Status: fixed)
• Slow WITH RECURSIVE "base_and_ancestors" AS ... https://gitlab.com/gitlab-org/gitlab-ce/issues/60628
• SeqScans on projects https://gitlab.com/gitlab-org/gitlab-ce/issues/60629

REPLICAS:

• Query with OFFSET (20% of all load) https://gitlab.com/gitlab-org/gitlab-ce/issues/61506