Skip to content

Send queries to fetch schema cache to replicas

Thong Kuah requested to merge send_schema_cache_queries_to_replicas into master

What does this MR do and why?

We have a CPU load issue when pods are created. Many pods all request schema information for tables either on boot, or on the first requests. All of these queries go to the primary, and are then cached in the schema cache.

It turns out that the schema queries always use the primary connection pool. That is the default behaviour, see ConnectionProxy#method_missing.

To alleviate this load issue, we configure schema_cache to use replica connection pools when possible.

Related issue: #398101 (closed)

Is it safe to use replicas to query schema information?:

Well, we happily rely on replicas for other queries - which means we implicitly trust the schema on replicas already :)

Screenshots or screen recordings

Before After
Screenshot_2023-03-30_at_3.37.00_PM Screenshot_2023-03-30_at_5.35.42_PM

How to set up and validate locally

  1. Set up DB load balancing in config/database.yml.

    load_balancing:
      hosts:
        - <same host as main:>
        - <same host as main:>
  2. gdk restart rails-web

  3. Navigate to 127.0.0.1:3000/

  4. Open the performance bar. Observe whether the SELECT a.attname,... queries is sent to primary, or replica.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Thong Kuah

Merge request reports