Investigate slowness in getState GraphQL query
Context
Issue originally raised by @stejacks-gitlab in #562413 (closed).
We had an incident today related to slowness in Api GraphQL for Canary, however, this is happening on main too. This has been going on for at least a month and appears to be slowly getting worse.
GraphQL for Canary:
GraphQL for Main:
There are two endpoints that were the biggest contributors to the slowness, getPipelineHeaderData and getState.
This has paged the EOC 19 times in the past 30 days.
This ticket was created from INC-3368 and was automatically exported by incident.io
Goal
We should investigate why so many of these are taking significantly more than 1 second.
- getState logs: https://log.gprd.gitlab.net/app/r/s/OJOOh
Out of scope
getPipelineHeaderData will be investigated in #562413 (closed).
Edited by François Rosé