Rails logs report that GraphqlChannel#subscribe uses a huge percentage of Postgres statement time on the Main Primary
GraphqlChannel#subscribe is a part of the GraphQL subscription service provided via Websockets.
It is mostly used in a limited number of places: issue labels (issuableLabelsUpdatedEE), issue epic changes (issuableEpicUpdated), merge request reviewer changes (mergeRequestReviewersUpdated), etc.
Along with other metrics, GraphqlChannel#subscribe reports the duration it spends in statements and transactions against our main postgres primary.
According to these logs, a huge percentage of all statement time on the postgres primary is devoted to handling these GraphqlChannel#subscribe requests.
This chart breaks the usage down across all rails controllers, using a sample of daily traffic over the past ~100 days.
Over the same period of time about ~1.4% of all requests were to this endpoint (source)
Timeseries
Looking back over the past ~100 days, we see that GraphqlChannel#subscribe is reporting major usage on our Main Postgres Primary for the entire time, with growth relative to other endpoints consuming large amounts of resources on the Main Postgres Primary.
Histogram of Reported Latencies
The durations being reported are very generally relatively short: in the region of about 5ms, however the volume of requests to this endpoint leads to a large volume in aggregate.
What next?
- Verify that the metric as reported is correct
- Investigate why this endpoint is using the primary and not a replica?
- Configure this endpoint to direct queries to replicas instead of primary
GraphQL subscriptions originally introduced in !59834 (merged) cc @engwan