Skip to content

Update strategies used for DB count approximation for read-only DB

Catalin Irimie requested to merge cat-approximate-counts-readonly-dbs into master

What does this MR do and why?

With the Tablesample and Reltuple strategies, they both attempt transactions to stick to the primary DB when load-balancing is used, which won't work with the loadbalancer enabled by default on read-only databases (such as Geo), as there is no writable primary DB.

Reasoning, on a read-only DB they rely on tuple stats to estimate the table sizes, which only works on the primary, writable DB:

[7] pry(main)> ::Gitlab::Database::Count.approximate_counts([Project, User, Group], strategies: [::Gitlab::Database::Count::TablesampleCountStrategy]).values
  TRANSACTION (0.3ms)  BEGIN /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
  Gitlab::Database::PgClass Load (5.0ms)  SELECT pg_class.relname AS table_name, reltuples::bigint AS estimate FROM "pg_class" LEFT JOIN pg_stat_user_tables ON pg_stat_user_tables.relid = pg_class.oid WHERE "pg_class"."relname" IN ('projects', 'users') AND (schemaname = current_schema()) /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
  TRANSACTION (0.3ms)  COMMIT /*application:console,db_config_name:main,line:/lib/gitlab/database.rb:287:in `commit'*/
   (1.7ms)  SELECT (COUNT(*)*0.002)::integer AS count
FROM projects
TABLESAMPLE SYSTEM (50000.0)
REPEATABLE (0)

 /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
=> []
[8] pry(main)> ::Gitlab::Database::Count.approximate_counts([Project, User, Group], strategies: [::Gitlab::Database::Count::ReltuplesCountStrategy]).values
  TRANSACTION (0.5ms)  BEGIN /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
  Gitlab::Database::PgClass Load (19.0ms)  SELECT pg_class.relname AS table_name, reltuples::bigint AS estimate FROM "pg_class" LEFT JOIN pg_stat_user_tables ON pg_stat_user_tables.relid = pg_class.oid WHERE "pg_class"."relname" IN ('projects', 'users') AND (schemaname = current_schema()) AND (last_vacuum > '2021-10-07 09:07:48.317002' OR last_autovacuum > '2021-10-07 09:07:48.317002' OR last_analyze > '2021-10-07 09:07:48.317002' OR last_autoanalyze > '2021-10-07 09:07:48.317002') /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
  TRANSACTION (0.6ms)  COMMIT /*application:console,db_config_name:main,line:/lib/gitlab/database.rb:287:in `commit'*/
=> []
[9] pry(main)> ::Gitlab::Database::Count.approximate_counts([Project, User, Group], strategies: [::Gitlab::Database::Count::ExactCountStrategy]).values
   (1.3ms)  SELECT COUNT(*) FROM "projects" /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
   (1.1ms)  SELECT COUNT(*) FROM "users" /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
   (0.9ms)  SELECT COUNT(*) FROM "namespaces" WHERE "namespaces"."type" = 'Group' /*application:console,db_config_name:main,line:/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'*/
=> [21, 98, 9]

Fixes #342083 (closed)

Screenshots or screen recordings

Before (just a 500 error in non-dev environments):

image

After, working as usual:

image

How to set up and validate locally

On a GDK + Geo setup (or through a way to make your primary DB read-only), go to /admin/ and /admin/users - they will 500 without this change.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Catalin Irimie

Merge request reports