postgres-checkup report for gitlab_production: 2019-11-11 (auto-generated)

PostgreSQL Checkup. Project: 'gitlab_production'. Database: 'gitlabhq_production'

Epoch number: '20191111001'

NOTICE: while most reports describe the “current database”, some of them may contain cluster-wide information describing all databases in the cluster.

Last modified at: 2019-11-11 13:49:36 +0000

A002 Version Information
A003 Postgres Settings
A004 Cluster Information
A005 Extensions
A006 Postgres Setting Deviations
A007 Altered Settings
D004 pg_stat_statements and pg_stat_kcache Settings
F001 Autovacuum: Current Settings
F002 Autovacuum: Transaction ID Wraparound Check
F003 Autovacuum: Dead Tuples
F004 Autovacuum: Heap Bloat (Estimated)
F005 Autovacuum: Btree Index Bloat (Estimated)
F008 Autovacuum: Resource Usage
G001 Memory-related Settings
G002 Connections and Current Activity
G003 Timeouts, Locks, Deadlocks
H001 Invalid Indexes
H002 Unused and Redundant Indexes
H003 Non-indexed Foreign Keys
K001 Globally Aggregated Query Metrics
K002 Workload Type ("The First Word" Analysis)
K003 Top-20 Queries by total_time
L001 Table Sizes
L003 Integer (int2, int4) Out-of-range Risks in PKs

Table of contents

A002 Version Information

Observations

Data collected: 2019-11-11 13:46:43 +0000 UTC

Master (`10.220.16.106`)

PostgreSQL 9.6.14 on x86_64-pc-linux-gnu (Ubuntu 9.6.14-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica servers:

Replica (`10.220.16.101`)

Nothing found

Replica (`10.220.16.102`)

PostgreSQL 9.6.14 on x86_64-pc-linux-gnu (Ubuntu 9.6.14-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica (`10.220.16.103`)

PostgreSQL 9.6.14 on x86_64-pc-linux-gnu (Ubuntu 9.6.14-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica (`10.220.16.104`)

PostgreSQL 9.6.14 on x86_64-pc-linux-gnu (Ubuntu 9.6.14-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica (`10.220.16.105`)

PostgreSQL 9.6.14 on x86_64-pc-linux-gnu (Ubuntu 9.6.14-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica (`10.220.16.107`)

PostgreSQL 9.6.15 on x86_64-pc-linux-gnu (Ubuntu 9.6.15-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica (`10.220.16.108`)

PostgreSQL 9.6.15 on x86_64-pc-linux-gnu (Ubuntu 9.6.15-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Replica (`10.220.16.109`)

PostgreSQL 9.6.15 on x86_64-pc-linux-gnu (Ubuntu 9.6.15-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit

Conclusions

[P2] Postgres minor version on the master and replica(s) differ. Nodes 10.220.16.108, 10.220.16.109, 10.220.16.102, 10.220.16.103, 10.220.16.104, 10.220.16.105, 10.220.16.106, 10.220.16.107 use Postgres 9.6.15, 9.6.14.
Postgres major version being used is 9.6 and it is currently supported by PostgreSQL Global Development Group. End of life is scheduled 2021-11-11. It means that in case of bugs and security issues, updates (new minor versions) with fixes will be released and available for use. Read more: Versioning Policy.
9.6.15 is the most up-to-date Postgres minor version in the branch 9.6.
[P2] The minor version being used (9.6.14) is not up-to-date (the newest version: 9.6.15). See the full list of changes between 9.6.14 and 9.6.15.

Recommendations

[P2] Please upgrade Postgres so its versions on all nodes match.
[P3] Consider upgrading to the newest major version: 11. It has a lot of new features and improvements.
For more information about minor and major upgrades see:
- Official documentation: https://www.postgresql.org/docs/current/upgrading.html
- Major-version upgrading with minimal downtime (depesz.com)
- Upgrading PostgreSQL on AWS RDS with minimum or zero downtime
- Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (2ndQuadrant.com)
- Updating a 50 terabyte PostgreSQL database
[P2] Consider performing upgrade to the newest minor version: 9.6.15.

Table of contents

A003 Postgres Settings

Observations

Data collected: 2019-11-11 13:46:43 +0000 UTC

Master (`10.220.16.106`)

▼ Category	Setting	Value	Unit	Pretty value
Autovacuum	autovacuum	on
Autovacuum	autovacuum_analyze_scale_factor	0.005
Autovacuum	autovacuum_analyze_threshold	50
Autovacuum	autovacuum_freeze_max_age	200000000
Autovacuum	autovacuum_max_workers	6
Autovacuum	autovacuum_multixact_freeze_max_age	400000000
Autovacuum	autovacuum_naptime	30	s
Autovacuum	autovacuum_vacuum_cost_delay	5	ms
Autovacuum	autovacuum_vacuum_cost_limit	6000
Autovacuum	autovacuum_vacuum_scale_factor	0.005
Autovacuum	autovacuum_vacuum_threshold	50
Client Connection Defaults / Locale and Formatting	DateStyle	ISO, MDY
Client Connection Defaults / Locale and Formatting	IntervalStyle	postgres
Client Connection Defaults / Locale and Formatting	TimeZone	GMT
Client Connection Defaults / Locale and Formatting	client_encoding	UTF8
Client Connection Defaults / Locale and Formatting	default_text_search_config	pg_catalog.english
Client Connection Defaults / Locale and Formatting	extra_float_digits	0
Client Connection Defaults / Locale and Formatting	lc_collate	C.UTF-8
Client Connection Defaults / Locale and Formatting	lc_ctype	C.UTF-8
Client Connection Defaults / Locale and Formatting	lc_messages	C
Client Connection Defaults / Locale and Formatting	lc_monetary	C
Client Connection Defaults / Locale and Formatting	lc_numeric	C
Client Connection Defaults / Locale and Formatting	lc_time	C
Client Connection Defaults / Locale and Formatting	server_encoding	UTF8
Client Connection Defaults / Locale and Formatting	timezone_abbreviations	Default
Client Connection Defaults / Other Defaults	dynamic_library_path	$libdir
Client Connection Defaults / Other Defaults	gin_fuzzy_search_limit	0
Client Connection Defaults / Other Defaults	tcp_keepalives_count	9
Client Connection Defaults / Other Defaults	tcp_keepalives_idle	300	s
Client Connection Defaults / Other Defaults	tcp_keepalives_interval	75	s
Client Connection Defaults / Shared Library Preloading	local_preload_libraries
Client Connection Defaults / Shared Library Preloading	session_preload_libraries
Client Connection Defaults / Shared Library Preloading	shared_preload_libraries	pg_stat_statements
Client Connection Defaults / Statement Behavior	bytea_output	hex
Client Connection Defaults / Statement Behavior	check_function_bodies	on
Client Connection Defaults / Statement Behavior	client_min_messages	notice
Client Connection Defaults / Statement Behavior	default_tablespace
Client Connection Defaults / Statement Behavior	default_transaction_deferrable	off
Client Connection Defaults / Statement Behavior	default_transaction_isolation	read committed
Client Connection Defaults / Statement Behavior	default_transaction_read_only	off
Client Connection Defaults / Statement Behavior	gin_pending_list_limit	4096	kB	4.00 MiB
Client Connection Defaults / Statement Behavior	idle_in_transaction_session_timeout	30000	ms
Client Connection Defaults / Statement Behavior	lock_timeout	0	ms
Client Connection Defaults / Statement Behavior	search_path	"$user", public
Client Connection Defaults / Statement Behavior	session_replication_role	origin
Client Connection Defaults / Statement Behavior	statement_timeout	15000	ms
Client Connection Defaults / Statement Behavior	temp_tablespaces
Client Connection Defaults / Statement Behavior	transaction_deferrable	off
Client Connection Defaults / Statement Behavior	transaction_isolation	read committed
Client Connection Defaults / Statement Behavior	transaction_read_only	off
Client Connection Defaults / Statement Behavior	vacuum_freeze_min_age	50000000
Client Connection Defaults / Statement Behavior	vacuum_freeze_table_age	150000000
Client Connection Defaults / Statement Behavior	vacuum_multixact_freeze_min_age	5000000
Client Connection Defaults / Statement Behavior	vacuum_multixact_freeze_table_age	150000000
Client Connection Defaults / Statement Behavior	xmlbinary	base64
Client Connection Defaults / Statement Behavior	xmloption	content
Connections and Authentication / Connection Settings	bonjour	off
Connections and Authentication / Connection Settings	bonjour_name
Connections and Authentication / Connection Settings	listen_addresses	0.0.0.0
Connections and Authentication / Connection Settings	max_connections	300
Connections and Authentication / Connection Settings	port	5432
Connections and Authentication / Connection Settings	superuser_reserved_connections	3
Connections and Authentication / Connection Settings	unix_socket_directories	/var/opt/gitlab/postgresql
Connections and Authentication / Connection Settings	unix_socket_group
Connections and Authentication / Connection Settings	unix_socket_permissions	0777
Connections and Authentication / Security and Authentication	authentication_timeout	60	s
Connections and Authentication / Security and Authentication	db_user_namespace	off
Connections and Authentication / Security and Authentication	krb_caseins_users	off
Connections and Authentication / Security and Authentication	krb_server_keyfile	FILE:/etc/postgresql-common/krb5.keytab
Connections and Authentication / Security and Authentication	password_encryption	on
Connections and Authentication / Security and Authentication	row_security	on
Connections and Authentication / Security and Authentication	ssl	on
Connections and Authentication / Security and Authentication	ssl_ca_file
Connections and Authentication / Security and Authentication	ssl_cert_file	/var/opt/gitlab/postgresql/server.crt
Connections and Authentication / Security and Authentication	ssl_ciphers	HIGH:MEDIUM:+3DES:!aNULL:!SSLv3:!TLSv1
Connections and Authentication / Security and Authentication	ssl_crl_file
Connections and Authentication / Security and Authentication	ssl_ecdh_curve	prime256v1
Connections and Authentication / Security and Authentication	ssl_key_file	/var/opt/gitlab/postgresql/server.key
Connections and Authentication / Security and Authentication	ssl_prefer_server_ciphers	on
Customized Options	pg_stat_statements.max	5000
Customized Options	pg_stat_statements.save	on
Customized Options	pg_stat_statements.track	top
Customized Options	pg_stat_statements.track_utility	on
Developer Options	allow_system_table_mods	off
Developer Options	ignore_checksum_failure	off
Developer Options	ignore_system_indexes	off
Developer Options	post_auth_delay	0	s
Developer Options	pre_auth_delay	0	s
Developer Options	trace_notify	off
Developer Options	trace_recovery_messages	log
Developer Options	trace_sort	off
Developer Options	zero_damaged_pages	off
Error Handling	data_sync_retry	off
Error Handling	exit_on_error	off
Error Handling	restart_after_crash	on
File Locations	config_file	/var/opt/gitlab/postgresql/postgresql.conf
File Locations	data_directory	/var/opt/gitlab/postgresql/data
File Locations	external_pid_file
File Locations	hba_file	/var/opt/gitlab/postgresql/data/pg_hba.conf
File Locations	ident_file	/var/opt/gitlab/postgresql/pg_ident.conf
Lock Management	deadlock_timeout	5000	ms
Lock Management	max_locks_per_transaction	128
Lock Management	max_pred_locks_per_transaction	64
Preset Options	block_size	8192
Preset Options	data_checksums	off
Preset Options	debug_assertions	off
Preset Options	integer_datetimes	on
Preset Options	max_function_args	100
Preset Options	max_identifier_length	63
Preset Options	max_index_keys	32
Preset Options	segment_size	131072	8kB	1.00 GiB
Preset Options	server_version	9.6.14
Preset Options	server_version_num	90614
Preset Options	wal_block_size	8192
Preset Options	wal_segment_size	2048	8kB	16.00 MiB
Process Title	cluster_name	pg-ha-cluster
Process Title	update_process_title	on
Query Tuning / Genetic Query Optimizer	geqo	on
Query Tuning / Genetic Query Optimizer	geqo_effort	5
Query Tuning / Genetic Query Optimizer	geqo_generations	0
Query Tuning / Genetic Query Optimizer	geqo_pool_size	0
Query Tuning / Genetic Query Optimizer	geqo_seed	0
Query Tuning / Genetic Query Optimizer	geqo_selection_bias	2
Query Tuning / Genetic Query Optimizer	geqo_threshold	12
Query Tuning / Other Planner Options	constraint_exclusion	partition
Query Tuning / Other Planner Options	cursor_tuple_fraction	0.1
Query Tuning / Other Planner Options	default_statistics_target	1000
Query Tuning / Other Planner Options	force_parallel_mode	off
Query Tuning / Other Planner Options	from_collapse_limit	8
Query Tuning / Other Planner Options	join_collapse_limit	8
Query Tuning / Planner Cost Constants	cpu_index_tuple_cost	0.005
Query Tuning / Planner Cost Constants	cpu_operator_cost	0.0025
Query Tuning / Planner Cost Constants	cpu_tuple_cost	0.01
Query Tuning / Planner Cost Constants	effective_cache_size	43352064	8kB	330.75 GiB
Query Tuning / Planner Cost Constants	min_parallel_relation_size	1024	8kB	8.00 MiB
Query Tuning / Planner Cost Constants	parallel_setup_cost	1000
Query Tuning / Planner Cost Constants	parallel_tuple_cost	0.1
Query Tuning / Planner Cost Constants	random_page_cost	1.5
Query Tuning / Planner Cost Constants	seq_page_cost	4
Query Tuning / Planner Method Configuration	enable_bitmapscan	on
Query Tuning / Planner Method Configuration	enable_hashagg	on
Query Tuning / Planner Method Configuration	enable_hashjoin	on
Query Tuning / Planner Method Configuration	enable_indexonlyscan	on
Query Tuning / Planner Method Configuration	enable_indexscan	on
Query Tuning / Planner Method Configuration	enable_material	on
Query Tuning / Planner Method Configuration	enable_mergejoin	on
Query Tuning / Planner Method Configuration	enable_nestloop	on
Query Tuning / Planner Method Configuration	enable_seqscan	on
Query Tuning / Planner Method Configuration	enable_sort	on
Query Tuning / Planner Method Configuration	enable_tidscan	on
Replication	track_commit_timestamp	off
Replication / Master Server	synchronous_standby_names
Replication / Master Server	vacuum_defer_cleanup_age	0
Replication / Sending Servers	max_replication_slots	32
Replication / Sending Servers	max_wal_senders	32
Replication / Sending Servers	wal_keep_segments	2048
Replication / Sending Servers	wal_sender_timeout	60000	ms
Replication / Standby Servers	hot_standby	on
Replication / Standby Servers	hot_standby_feedback	on
Replication / Standby Servers	max_standby_archive_delay	30000	ms
Replication / Standby Servers	max_standby_streaming_delay	30000	ms
Replication / Standby Servers	wal_receiver_status_interval	10	s
Replication / Standby Servers	wal_receiver_timeout	60000	ms
Replication / Standby Servers	wal_retrieve_retry_interval	5000	ms
Reporting and Logging / What to Log	application_name	checkup
Reporting and Logging / What to Log	debug_pretty_print	on
Reporting and Logging / What to Log	debug_print_parse	off
Reporting and Logging / What to Log	debug_print_plan	off
Reporting and Logging / What to Log	debug_print_rewritten	off
Reporting and Logging / What to Log	log_autovacuum_min_duration	0	ms
Reporting and Logging / What to Log	log_checkpoints	on
Reporting and Logging / What to Log	log_connections	off
Reporting and Logging / What to Log	log_disconnections	off
Reporting and Logging / What to Log	log_duration	off
Reporting and Logging / What to Log	log_error_verbosity	default
Reporting and Logging / What to Log	log_hostname	off
Reporting and Logging / What to Log	log_line_prefix	%m [%p, %x]: [%l-1] user=%u, db=%d, app=%a, client=%h
Reporting and Logging / What to Log	log_lock_waits	on
Reporting and Logging / What to Log	log_replication_commands	off
Reporting and Logging / What to Log	log_statement	ddl
Reporting and Logging / What to Log	log_temp_files	0	kB	0.00 bytes
Reporting and Logging / What to Log	log_timezone	GMT
Reporting and Logging / When to Log	log_min_duration_statement	1000	ms
Reporting and Logging / When to Log	log_min_error_statement	error
Reporting and Logging / When to Log	log_min_messages	warning
Reporting and Logging / Where to Log	event_source	PostgreSQL
Reporting and Logging / Where to Log	log_destination	stderr
Reporting and Logging / Where to Log	log_directory	/var/log/gitlab/postgresql
Reporting and Logging / Where to Log	log_file_mode	0640
Reporting and Logging / Where to Log	log_filename	postgresql.log
Reporting and Logging / Where to Log	log_rotation_age	0	min
Reporting and Logging / Where to Log	log_rotation_size	0	kB	0.00 bytes
Reporting and Logging / Where to Log	log_truncate_on_rotation	off
Reporting and Logging / Where to Log	logging_collector	on
Reporting and Logging / Where to Log	syslog_facility	local0
Reporting and Logging / Where to Log	syslog_ident	postgres
Reporting and Logging / Where to Log	syslog_sequence_numbers	on
Reporting and Logging / Where to Log	syslog_split_messages	on
Resource Usage / Asynchronous Behavior	backend_flush_after	0	8kB	0.00 bytes
Resource Usage / Asynchronous Behavior	effective_io_concurrency	100
Resource Usage / Asynchronous Behavior	max_parallel_workers_per_gather	0
Resource Usage / Asynchronous Behavior	max_worker_processes	8
Resource Usage / Asynchronous Behavior	old_snapshot_threshold	-1	min
Resource Usage / Background Writer	bgwriter_delay	200	ms
Resource Usage / Background Writer	bgwriter_flush_after	64	8kB	512.00 KiB
Resource Usage / Background Writer	bgwriter_lru_maxpages	100
Resource Usage / Background Writer	bgwriter_lru_multiplier	2
Resource Usage / Cost-Based Vacuum Delay	vacuum_cost_delay	0	ms
Resource Usage / Cost-Based Vacuum Delay	vacuum_cost_limit	200
Resource Usage / Cost-Based Vacuum Delay	vacuum_cost_page_dirty	20
Resource Usage / Cost-Based Vacuum Delay	vacuum_cost_page_hit	1
Resource Usage / Cost-Based Vacuum Delay	vacuum_cost_page_miss	10
Resource Usage / Disk	temp_file_limit	-1	kB
Resource Usage / Kernel Resources	max_files_per_process	1000
Resource Usage / Memory	autovacuum_work_mem	-1	kB
Resource Usage / Memory	dynamic_shared_memory_type	posix
Resource Usage / Memory	huge_pages	try
Resource Usage / Memory	maintenance_work_mem	2097152	kB	2.00 GiB
Resource Usage / Memory	max_prepared_transactions	0
Resource Usage / Memory	max_stack_depth	2048	kB	2.00 MiB
Resource Usage / Memory	replacement_sort_tuples	150000
Resource Usage / Memory	shared_buffers	14450688	8kB	110.25 GiB
Resource Usage / Memory	temp_buffers	1024	8kB	8.00 MiB
Resource Usage / Memory	track_activity_query_size	1024
Resource Usage / Memory	work_mem	102400	kB	100.00 MiB
Statistics / Monitoring	log_executor_stats	off
Statistics / Monitoring	log_parser_stats	off
Statistics / Monitoring	log_planner_stats	off
Statistics / Monitoring	log_statement_stats	off
Statistics / Query and Index Statistics Collector	stats_temp_directory	pg_stat_tmp
Statistics / Query and Index Statistics Collector	track_activities	on
Statistics / Query and Index Statistics Collector	track_counts	on
Statistics / Query and Index Statistics Collector	track_functions	none
Statistics / Query and Index Statistics Collector	track_io_timing	on
Version and Platform Compatibility / Other Platforms and Clients	transform_null_equals	off
Version and Platform Compatibility / Previous PostgreSQL Versions	array_nulls	on
Version and Platform Compatibility / Previous PostgreSQL Versions	backslash_quote	safe_encoding
Version and Platform Compatibility / Previous PostgreSQL Versions	default_with_oids	off
Version and Platform Compatibility / Previous PostgreSQL Versions	escape_string_warning	on
Version and Platform Compatibility / Previous PostgreSQL Versions	lo_compat_privileges	off
Version and Platform Compatibility / Previous PostgreSQL Versions	operator_precedence_warning	off
Version and Platform Compatibility / Previous PostgreSQL Versions	quote_all_identifiers	off
Version and Platform Compatibility / Previous PostgreSQL Versions	sql_inheritance	on
Version and Platform Compatibility / Previous PostgreSQL Versions	standard_conforming_strings	on
Version and Platform Compatibility / Previous PostgreSQL Versions	synchronize_seqscans	on
Write-Ahead Log / Archiving	archive_command	/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push %p
Write-Ahead Log / Archiving	archive_mode	on
Write-Ahead Log / Archiving	archive_timeout	60	s
Write-Ahead Log / Checkpoints	checkpoint_completion_target	0.7
Write-Ahead Log / Checkpoints	checkpoint_flush_after	32	8kB	256.00 KiB
Write-Ahead Log / Checkpoints	checkpoint_timeout	300	s
Write-Ahead Log / Checkpoints	checkpoint_warning	30	s
Write-Ahead Log / Checkpoints	max_wal_size	320	16MB	5.00 GiB
Write-Ahead Log / Checkpoints	min_wal_size	5	16MB	80.00 MiB
Write-Ahead Log / Settings	commit_delay	0
Write-Ahead Log / Settings	commit_siblings	5
Write-Ahead Log / Settings	fsync	on
Write-Ahead Log / Settings	full_page_writes	on
Write-Ahead Log / Settings	synchronous_commit	on
Write-Ahead Log / Settings	wal_buffers	2048	8kB	16.00 MiB
Write-Ahead Log / Settings	wal_compression	off
Write-Ahead Log / Settings	wal_level	replica
Write-Ahead Log / Settings	wal_log_hints	on
Write-Ahead Log / Settings	wal_sync_method	fdatasync
Write-Ahead Log / Settings	wal_writer_delay	200	ms
Write-Ahead Log / Settings	wal_writer_flush_after	128	8kB	1.00 MiB

Table of contents

A004 Cluster Information

Observations

Data collected: 2019-11-11 13:47:10 +0000 UTC

▼ Indicator	10.220.16.106	10.220.16.102	10.220.16.103	10.220.16.104	10.220.16.105	10.220.16.107	10.220.16.108	10.220.16.109
Config file	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf	/var/opt/gitlab/postgresql/postgresql.conf
Role	Master	Replica (delay: 00:00:00; paused: false)	Replica (delay: 00:00:00; paused: false)	Replica (delay: 00:00:00; paused: false)	Replica (delay: 00:00:00; paused: false)	Replica (delay: 00:00:00; paused: false)	Replica (delay: 00:00:00; paused: false)	Replica (delay: 00:00:00; paused: false)
Replicas	async/streaming: 10.220.16.105, 10.220.16.102, 10.220.16.104, 10.220.16.103, 10.220.16.108, 10.220.16.112, 10.220.16.111, 10.220.16.109, 10.220.16.107
Started At	2019-10-28 16:07:17+00	2019-10-29 09:21:51+00	2019-10-29 09:21:51+00	2019-10-29 09:21:51+00	2019-10-29 09:21:50+00	2019-10-29 17:48:07+00	2019-10-29 09:21:51+00	2019-10-29 09:21:53+00
Uptime	13 days 21:32:32	13 days 04:14:00	13 days 04:14:38	13 days 04:15:29	13 days 04:16:35	12 days 19:54:25	13 days 04:22:48	13 days 04:25:19
Checkpoints	18443	189975	189943	189394	189215	177074	187590	187693
Forced Checkpoints	90.7%	99.6%	99.6%	99.6%	99.6%	99.6%	99.6%	99.6%
Checkpoint MB/sec	12.544240	14.000570	13.998910	13.999882	14.000570	13.670592	14.000040	14.001197
Database Name	gitlabhq_production	gitlabhq_production	gitlabhq_production	gitlabhq_production	gitlabhq_production	gitlabhq_production	gitlabhq_production	gitlabhq_production
Database Size	4502 GB	4502 GB	4502 GB	4502 GB	4502 GB	4502 GB	4502 GB	4502 GB
Stats Since	2019-10-28 16:07:35+00	2019-10-29 09:21:59+00	2019-10-29 09:22:00+00	2019-10-29 09:22:00+00	2019-10-29 09:21:59+00	2019-10-29 17:48:34+00	2019-10-29 09:22:05+00	2019-10-29 09:22:06+00
Stats Age	13 days 21:32:14	13 days 04:13:52	13 days 04:14:29	13 days 04:15:20	13 days 04:16:27	12 days 19:53:58	13 days 04:22:33	13 days 04:25:05
Cache Effectiveness	99.35%	99.82%	99.81%	99.82%	99.82%	99.84%	99.82%	99.81%
Successful Commits	99.64%	100.00%	100.00%	100.00%	100.00%	100.00%	100.00%	100.00%
Conflicts	4	0	0	0	0	0	0	0
Temp Files: total size	121 GB	803 GB	821 GB	814 GB	811 GB	787 GB	785 GB	834 GB
Temp Files: total number of files	2439	5692	5774	5790	5868	4984	5638	5892
Temp Files: total number of files per day	174	437	444	445	451	383	433	453
Temp Files: avg file size	51 MB	144 MB	146 MB	144 MB	142 MB	162 MB	143 MB	145 MB
Deadlocks	217	0	0	0	0	0	0	0
Deadlocks per day	15	0	0	0	0	0	0	0

Databases sizes

Database	▼ Size
`gitlabhq_production`	4.40 TiB
`postgres`	6.99 MiB
`template0`	6.91 MiB
`template1`	6.91 MiB

Conclusions

Recommendations

Table of contents

A005 Extensions

Observations

Data collected: 2019-11-11 13:47:11 +0000 UTC

Master (`10.220.16.106`)

▼ Database	Extension name	Installed version	Default version
`gitlabhq_production`	`btree_gin`	1.0	1.0
`gitlabhq_production`	`dblink`	1.2	1.2
`gitlabhq_production`	`pageinspect`	1.5	1.5
`gitlabhq_production`	`pg_buffercache`	1.2	1.2
`gitlabhq_production`	`pg_repack`	1.4.4	1.4.4
`gitlabhq_production`	`pg_stat_statements`	1.4	1.4
`gitlabhq_production`	`pg_trgm`	1.3	1.3
`gitlabhq_production`	`pgstattuple`	1.4	1.4
`gitlabhq_production`	`plpgsql`	1.0	1.0

Conclusions

Recommendations

Table of contents

A006 Postgres Setting Deviations

Observations

Data collected: 2019-11-11 13:47:12 +0000 UTC

Settings (pg_settings) that Differ

▼ Setting	10.220.16.106	10.220.16.101	10.220.16.102	10.220.16.103	10.220.16.104	10.220.16.105	10.220.16.107	10.220.16.108	10.220.16.109
server_version	9.6.14		9.6.14	9.6.14	9.6.14	9.6.14	9.6.15	9.6.15	9.6.15
server_version_num	90614		90614	90614	90614	90614	90615	90615	90615

Configs(pg_config) that differ

▼ Config	10.220.16.106	10.220.16.101	10.220.16.102	10.220.16.103	10.220.16.104	10.220.16.105	10.220.16.107	10.220.16.108	10.220.16.109
CONFIGURE	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.14-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'		'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.14-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.14-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.14-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.14-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.15-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.15-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'	'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python' '--mandir=/usr/share/postgresql/9.6/man' '--docdir=/usr/share/doc/postgresql-doc-9.6' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/9.6' '--bindir=/usr/lib/postgresql/9.6/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -pie -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' 'DPKG_VERSION=9.6.15-1.pgdg16.04+1' 'DPKG_VENDOR=Ubuntu' '--with-gssapi' '--with-ldap' '--with-includes=/usr/include/mit-krb5' '--with-libs=/usr/lib/mit-krb5' '--with-libs=/usr/lib/x86_64-linux-gnu/mit-krb5' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
VERSION	PostgreSQL 9.6.14		PostgreSQL 9.6.14	PostgreSQL 9.6.14	PostgreSQL 9.6.14	PostgreSQL 9.6.14	PostgreSQL 9.6.15	PostgreSQL 9.6.15	PostgreSQL 9.6.15

Conclusions

Recommendations

Table of contents

A007 Altered Settings

Observations

Data collected: 2019-11-11 13:47:44 +0000 UTC

Master (`10.220.16.106`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	5	autovacuum_naptime idle_in_transaction_session_timeout log_min_duration_statement track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	46	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica servers:

Replica (`10.220.16.101`)

No data

Replica (`10.220.16.102`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica (`10.220.16.103`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica (`10.220.16.104`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica (`10.220.16.105`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica (`10.220.16.107`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica (`10.220.16.108`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Replica (`10.220.16.109`)

Source	Settings count	Changed settings
/var/opt/gitlab/postgresql/data/postgresql.auto.conf	4	autovacuum_naptime idle_in_transaction_session_timeout track_io_timing wal_keep_segments
/var/opt/gitlab/postgresql/postgresql.base.conf	2	autovacuum_freeze_max_age min_wal_size
/var/opt/gitlab/postgresql/postgresql.conf	47	archive_command archive_mode archive_timeout autovacuum_analyze_scale_factor autovacuum_max_workers autovacuum_vacuum_cost_delay autovacuum_vacuum_cost_limit autovacuum_vacuum_scale_factor checkpoint_completion_target checkpoint_timeout deadlock_timeout default_statistics_target default_text_search_config effective_cache_size effective_io_concurrency hot_standby_feedback lc_messages log_autovacuum_min_duration log_checkpoints log_destination log_directory log_file_mode log_filename log_line_prefix log_lock_waits log_min_duration_statement log_rotation_age log_rotation_size log_statement log_temp_files logging_collector maintenance_work_mem max_wal_size random_page_cost shared_buffers shared_preload_libraries ssl ssl_ca_file ssl_cert_file ssl_ciphers ssl_key_file tcp_keepalives_count tcp_keepalives_idle tcp_keepalives_interval track_activity_query_size unix_socket_directories work_mem
default	213

Conclusions

Recommendations

Table of contents

D004 pg_stat_statements and pg_stat_kcache Settings

Observations

Data collected: 2019-11-11 13:47:45 +0000 UTC

Master (`10.220.16.106`)

`pg_stat_statements` extension settings

Setting	Value	Type	Min value	Max value
pg_stat_statements.max	5000	integer	100	2147483647
pg_stat_statements.save	on	bool
pg_stat_statements.track	top	enum
pg_stat_statements.track_utility	on	bool

Conclusions

Recommendations

Table of contents

F001 Autovacuum: Current Settings

Observations

Data collected: 2019-11-11 13:47:45 +0000 UTC

Master (`10.220.16.106`)

▼ Setting name	Value	Unit	Pretty value
autovacuum	on
autovacuum_analyze_scale_factor	0.005
autovacuum_analyze_threshold	50
autovacuum_freeze_max_age	200000000
autovacuum_max_workers	6
autovacuum_multixact_freeze_max_age	400000000
autovacuum_naptime	30	s
autovacuum_vacuum_cost_delay	5	ms
autovacuum_vacuum_cost_limit	6000
autovacuum_vacuum_scale_factor	0.005
autovacuum_vacuum_threshold	50
autovacuum_work_mem	-1	kB
maintenance_work_mem	2097152	kB	2.00 GiB
vacuum_cost_delay	0	ms
vacuum_cost_limit	200
vacuum_cost_page_dirty	20
vacuum_cost_page_hit	1
vacuum_cost_page_miss	10
vacuum_defer_cleanup_age	0
vacuum_freeze_min_age	50000000
vacuum_freeze_table_age	150000000
vacuum_multixact_freeze_min_age	5000000
vacuum_multixact_freeze_table_age	150000000

Tuned tables

▼ Namespace	Relation	Options
pg_toast	pg_toast_33145	autovacuum_enabled=true
pg_toast	pg_toast_33153	autovacuum_enabled=true
pg_toast	pg_toast_33161	autovacuum_enabled=true
pg_toast	pg_toast_33208	autovacuum_enabled=true
pg_toast	pg_toast_33221	autovacuum_enabled=true
pg_toast	pg_toast_33229	autovacuum_enabled=true
pg_toast	pg_toast_33237	autovacuum_enabled=true
pg_toast	pg_toast_33248	autovacuum_enabled=true
pg_toast	pg_toast_33262	autovacuum_enabled=true
pg_toast	pg_toast_33272	autovacuum_enabled=true
pg_toast	pg_toast_33322	autovacuum_enabled=true
pg_toast	pg_toast_33356	autovacuum_enabled=true
pg_toast	pg_toast_33369	autovacuum_enabled=true
pg_toast	pg_toast_33383	autovacuum_enabled=true
pg_toast	pg_toast_33393	autovacuum_enabled=true
pg_toast	pg_toast_33400	autovacuum_enabled=true
pg_toast	pg_toast_33426	autovacuum_enabled=true
pg_toast	pg_toast_33439	autovacuum_enabled=true
pg_toast	pg_toast_33447	autovacuum_enabled=true
pg_toast	pg_toast_33462	autovacuum_enabled=true
pg_toast	pg_toast_33476	autovacuum_enabled=true
pg_toast	pg_toast_33497	autovacuum_enabled=true
pg_toast	pg_toast_33506	autovacuum_enabled=true
pg_toast	pg_toast_33514	autovacuum_enabled=true
pg_toast	pg_toast_33527	autovacuum_enabled=true
pg_toast	pg_toast_33541	autovacuum_enabled=true
pg_toast	pg_toast_33553	autovacuum_enabled=true
pg_toast	pg_toast_33568	autovacuum_enabled=true
pg_toast	pg_toast_33587	autovacuum_enabled=true
pg_toast	pg_toast_33597	autovacuum_enabled=true
pg_toast	pg_toast_33614	autovacuum_enabled=true
pg_toast	pg_toast_33625	autovacuum_enabled=true
pg_toast	pg_toast_33634	autovacuum_enabled=true
pg_toast	pg_toast_33642	autovacuum_enabled=true
pg_toast	pg_toast_33650	autovacuum_enabled=true
pg_toast	pg_toast_33659	autovacuum_enabled=true
pg_toast	pg_toast_33667	autovacuum_enabled=true
pg_toast	pg_toast_33675	autovacuum_enabled=true
pg_toast	pg_toast_33696	autovacuum_enabled=true
pg_toast	pg_toast_33706	autovacuum_enabled=true
pg_toast	pg_toast_33749	autovacuum_enabled=true
pg_toast	pg_toast_33756	autovacuum_enabled=true
pg_toast	pg_toast_33768	autovacuum_enabled=true
pg_toast	pg_toast_33776	autovacuum_enabled=true
pg_toast	pg_toast_33791	autovacuum_enabled=true
pg_toast	pg_toast_33801	autovacuum_enabled=true
pg_toast	pg_toast_33824	autovacuum_enabled=true
pg_toast	pg_toast_33835	autovacuum_enabled=true
pg_toast	pg_toast_33851	autovacuum_enabled=true
pg_toast	pg_toast_33863	autovacuum_enabled=true
pg_toast	pg_toast_33872	autovacuum_enabled=true
pg_toast	pg_toast_33885	autovacuum_enabled=true
pg_toast	pg_toast_33898	autovacuum_enabled=true
pg_toast	pg_toast_33909	autovacuum_enabled=true
pg_toast	pg_toast_33959	autovacuum_enabled=true
public	abuse_reports	autovacuum_enabled=true
public	appearances	autovacuum_enabled=true
public	application_settings	autovacuum_enabled=true
public	approval_merge_request_rules_users	autovacuum_enabled=on
public	approvals	autovacuum_enabled=true
public	approver_groups	autovacuum_enabled=true
public	approvers	autovacuum_enabled=on
public	audit_events	autovacuum_enabled=true
public	award_emoji	autovacuum_enabled=on
public	boards	autovacuum_enabled=true
public	broadcast_messages	autovacuum_enabled=true
public	ci_builds	autovacuum_enabled=true
public	ci_pipelines	autovacuum_enabled=true
public	ci_runner_projects	autovacuum_enabled=true
public	ci_runners	autovacuum_enabled=on
public	ci_trigger_requests	autovacuum_enabled=true
public	ci_triggers	autovacuum_enabled=true
public	ci_variables	autovacuum_enabled=true
public	deploy_keys_projects	autovacuum_enabled=true
public	deployments	autovacuum_enabled=on
public	emails	autovacuum_enabled=true
public	environments	autovacuum_enabled=true
public	forked_project_links	autovacuum_enabled=true
public	geo_nodes	autovacuum_enabled=true
public	historical_data	autovacuum_enabled=true
public	identities	autovacuum_enabled=true
public	index_statuses	autovacuum_enabled=true
public	issue_metrics	autovacuum_enabled=true
public	issues	autovacuum_enabled=true
public	keys	autovacuum_enabled=true
public	label_links	autovacuum_enabled=true
public	label_priorities	autovacuum_enabled=true
public	labels	autovacuum_enabled=on
public	ldap_group_links	autovacuum_enabled=true
public	lfs_objects	autovacuum_enabled=true
public	lfs_objects_projects	autovacuum_enabled=true
public	licenses	autovacuum_enabled=true
public	lists	autovacuum_enabled=true
public	members	autovacuum_enabled=true
public	merge_request_diffs	autovacuum_enabled=true
public	merge_request_metrics	autovacuum_enabled=true
public	merge_requests	autovacuum_enabled=true
public	merge_requests_closing_issues	autovacuum_enabled=true
public	milestones	autovacuum_enabled=on
public	namespaces	autovacuum_enabled=on
public	notes	autovacuum_enabled=true
public	notification_settings	autovacuum_enabled=on
public	oauth_access_grants	autovacuum_enabled=true
public	oauth_access_tokens	autovacuum_enabled=true
public	oauth_applications	autovacuum_enabled=true
public	pages_domains	autovacuum_enabled=true
public	path_locks	autovacuum_enabled=true
public	personal_access_tokens	autovacuum_enabled=true
public	project_authorizations	autovacuum_enabled=true
public	project_features	autovacuum_enabled=true
public	project_group_links	autovacuum_enabled=true
public	project_import_data	autovacuum_enabled=on
public	project_mirror_data	autovacuum_vacuum_cost_limit=10000 autovacuum_enabled=on
public	project_repository_states	autovacuum_enabled=on
public	projects	autovacuum_vacuum_cost_limit=7000 autovacuum_enabled=on
public	protected_branch_merge_access_levels	autovacuum_enabled=true
public	protected_branch_push_access_levels	autovacuum_enabled=true
public	protected_branches	autovacuum_enabled=true
public	push_rules	autovacuum_enabled=true
public	releases	autovacuum_enabled=on
public	remote_mirrors	autovacuum_enabled=true
public	sent_notifications	autovacuum_enabled=true
public	services	autovacuum_enabled=on
public	snippets	autovacuum_enabled=on
public	spam_logs	autovacuum_enabled=true
public	subscriptions	autovacuum_enabled=true
public	taggings	autovacuum_enabled=true
public	tags	autovacuum_enabled=true
public	todos	autovacuum_enabled=true
public	trending_projects	autovacuum_enabled=true
public	u2f_registrations	autovacuum_enabled=true
public	untracked_files_for_uploads	autovacuum_enabled=on
public	user_agent_details	autovacuum_enabled=true
public	users	autovacuum_enabled=on
public	users_star_projects	autovacuum_enabled=true
public	web_hooks	autovacuum_enabled=true

Replicas settings

Setting	10.220.16.101	10.220.16.102	10.220.16.103	10.220.16.104	10.220.16.105	10.220.16.107	10.220.16.108	10.220.16.109
hot_standby_feedback		on	on	on	on	on	on	on

Conclusions

Recommendations

All good, no recommendations here.

Table of contents

F002 Autovacuum: Transaction ID Wraparound Check

Observations

Data collected: 2019-11-11 13:40:10 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Databases

#	Database	▼ Age	Capacity used, %	datfrozenxid
1	`gitlabhq_production`	196572867	10.08	1686521332
2	`template0`	170103736	8.72	1712990463
3	`template1`	169990043	8.72	1713104156
4	`postgres`	165365212	8.48	1717728987

Tables in the observed database

The list is limited to 10 items. Total: 285.

#	Relation	Age	▼ Capacity used, %	rel_relfrozenxid	toast_relfrozenxid
1	`pg_catalog.pg_class`	196572867	10.08	1686521332	0
2	`board_assignees`	193779735	9.94	1689314464	0
3	`gpg_keys`	185923084	9.53	1809457241	1697171115
4	`alerts_service_data`	185749659	9.53	1697344540	1697347812
5	`lists`*	180873299	9.28	1702220900	0
6	`merge_request_diffs`*	180111866	9.24	1702982333	1814665565
7	`epics`	180178187	9.24	1817427296	1702916012
8	`services`*	179408750	9.2	1712931189	1703685449
9	`routes`	178921558	9.18	1704172641	1813088555
10	`uploads`	178452001	9.15	1704642198	1813686255

* This table has specific autovacuum settings. See 'F001 Autovacuum: Current settings'

Conclusions

Recommendations

All good, no recommendations here.

Table of contents

F003 Autovacuum: Dead Tuples

Observations

Data collected: 2019-11-11 13:40:12 +0000 UTC
Current database: gitlabhq_production
Stats reset: 13 days 21:33:00 ago (2019-10-28 16:07:35 +0000 UTC)

Master (`10.220.16.106`)

The list is limited to 20 items. Total: 191.

#	Relation	reltype	Since last autovacuum	Autovacuum Count	n_tup_ins	n_tup_upd	n_tup_del	pg_class.reltuples	n_live_tup	n_dead_tup	▼Dead Tuples Ratio, %
1	`project_mirror_data`*	r	00:05:10.508254	1723	34103	227398672	9576	2240281	2240284	230085	9.31
2	`namespace_aggregation_schedules`	r	00:00:15.216235	48926	5447916	0	5355681	16883	16883	361	2.09
3	`ci_pipeline_schedules`	r	00:09:08.737742	3701	1018	1015295	413	22580	22579	456	1.98
4	`pool_repositories`	r	06:16:27.087769	26	1070	3189	8	12360	12364	99	0.79
5	`epics`	r	01:35:31.607346	131	451	21110	62	12521	12524	96	0.76
6	`pg_toast.pg_toast_2619`	t	00:00:15.034585	16729	2551420	0	2551323	16131	16024	119	0.74
7	`external_pull_requests`	r	03:05:26.663814	50	3807	11267	221	11001	11030	75	0.68
8	`cluster_projects`	r	1 day 16:57:58.974179	7	1442	0	1023	19846	19874	119	0.6
9	`path_locks`*	r	3 days 13:48:37.555912	3	376	0	289	10851	10851	57	0.52
10	`project_group_links`*	r	02:41:56.211322	92	68710	455	66436	134876	134876	703	0.52
11	`board_group_recent_visits`	r	00:20:45.370291	189	1304	71180	191	24830	24830	122	0.49
12	`suggestions`	r	3 days 17:57:49.827849	5	11262	4535	1503	119170	119374	589	0.49
13	`cluster_platforms_kubernetes`	r	22:30:39.625082	10	1721	1239	1098	26109	26123	128	0.49
14	`ci_job_artifacts`	r	1 day 17:57:58.992942	11	13047299	13030905	3457324	277387552	277535013	1341110	0.48
15	`internal_ids`	r	00:44:48.051172	223	83244	8213162	20943	2734666	2734764	13133	0.48
16	`todos`*	r	2 days 21:39:24.036587	7	1194977	908279	324709	34406572	34409140	160053	0.46
17	`protected_branch_merge_access_levels`*	r	2 days 04:57:01.514759	4	359613	3739	212687	9005468	9014956	41283	0.46
18	`project_import_data`*	r	1 day 22:04:25.266509	8	24567	3583	21711	538521	538668	2452	0.45
19	`projects`*	r	00:43:33.516022	213	218689	12506756	65173	11085568	11085833	49362	0.44
20	`ci_runners`*	r	00:01:24.234291	7818	28858	15747092	20063	412885	412885	1793	0.43

* This table has specific autovacuum settings. See 'F001 Autovacuum: Current settings'

Conclusions

Recommendations

Table of contents

F004 Autovacuum: Heap Bloat (Estimated)

⚠ This report is based on estimations. The errors in bloat estimates may be significant (in some cases, up to 15% and even more). Use it only as an indicator of potential issues.

Observations

Data collected: 2019-11-11 13:40:13 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

The list is limited to 20 items. Total: 280.

#	Table	Real Size	▼ Estimated bloat	Est. bloat, bytes	Est. bloat factor	Est. bloat level, %	Live Data Size	Last vacuum	Fillfactor
	===== TOTAL =====	3.02 TiB	71.06 GiB	76,292,243,456	1.02	2.30	~2.95 TiB
1	`merge_request_diff_files`	1.35 TiB	34.65 GiB	37,195,161,600	1.03	2.51	~1.32 TiB	2019-11-08 16:26:07 (auto)	100
2	`ci_builds`*	441.94 GiB	12.57 GiB	13,491,453,952	1.03	2.84	~429.38 GiB	2019-11-11 03:42:18 (auto)	100
3	`merge_request_diff_commits`	537.16 GiB	7.76 GiB	8,326,127,616	1.01	1.44	~529.41 GiB	2019-11-11 02:23:38 (auto)	100
4	`audit_events`*	46.64 GiB	2.18 GiB	2,331,090,944	1.05	4.66	~44.47 GiB	2019-11-11 01:47:41 (auto)	100
5	`notes`*	142.07 GiB	2.16 GiB	2,309,193,728	1.02	1.51	~139.91 GiB	2019-11-08 17:38:20 (auto)	100
6	`ci_pipelines`*	18.37 GiB	1.45 GiB	1,553,899,520	1.09	7.88	~16.92 GiB	2019-11-11 12:52:08 (auto)	100
7	`sent_notifications`*	33.34 GiB	1.30 GiB	1,391,837,184	1.04	3.89	~32.04 GiB	2019-11-11 03:38:44 (auto)	100
8	`ci_job_artifacts`	41.46 GiB	1.04 GiB	1,116,569,600	1.03	2.51	~40.42 GiB	2019-11-09 19:42:14 (auto)	100
9	`merge_requests`*	15.95 GiB	861.49 MiB	903,331,840	1.06	5.28	~15.10 GiB	2019-11-11 12:10:22 (auto)	100
10	`merge_request_diffs`*	11.66 GiB	810.89 MiB	850,272,256	1.07	6.80	~10.87 GiB	2019-11-10 15:34:25 (auto)	100
11	`ci_stages`	13.21 GiB	806.97 MiB	846,159,872	1.06	5.97	~12.42 GiB	2019-11-11 11:17:46 (auto)	100
12	`ci_build_trace_sections`	108.61 GiB	629.62 MiB	660,201,472	1.01	0.57	~108.00 GiB	2019-11-11 01:56:58 (auto)	100
13	`deployments`*	5.77 GiB	523.60 MiB	549,027,840	1.10	8.87	~5.26 GiB	2019-11-09 08:53:32 (auto)	100
14	`projects`*	3.22 GiB	417.09 MiB	437,346,304	1.15	12.66	~2.81 GiB	2019-11-11 12:56:39 (auto)	100
15	`project_mirror_data`*	560.25 MiB	308.94 MiB	323,944,448	2.23	55.14	~251.31 MiB	2019-11-11 13:35:02 (auto)	100
16	`resource_label_events`	11.40 GiB	291.69 MiB	305,856,512	1.03	2.50	~11.12 GiB	2019-11-10 19:42:28 (auto)	100
17	`todos`*	3.22 GiB	275.08 MiB	288,432,128	1.09	8.36	~2.95 GiB	2019-11-08 16:00:49 (auto)	100
18	`lfs_objects`*	3.69 GiB	256.27 MiB	268,713,984	1.07	6.79	~3.44 GiB	2019-11-09 20:48:54 (auto)	100
19	`users`*	2.06 GiB	223.25 MiB	234,086,400	1.12	10.60	~1.84 GiB	2019-11-11 13:32:32 (auto)	100
20	`uploads`	2.28 GiB	186.08 MiB	195,108,864	1.09	7.99	~2.10 GiB	2019-11-07 17:21:28 (auto)	100

* This table has specific autovacuum settings. See 'F001 Autovacuum: Current settings'

Conclusions

The estimated table (heap) bloat in this DB is low, just ~2.30% (~71.06 GiB). No action is needed now. Keep watching it though.
[P1] The following 2 tables have significant size (>1 MiB) and bloat estimate > 90.00%:
- ci_build_trace_chunks: size 4.34 MiB, can be reduced 69.38 times, by ~4.28 MiB (~98.56%)
- appearances: size 1.37 MiB, can be reduced 175.00 times, by ~1.36 MiB (~99.43%)
[P2] There are 6 tables with size > 1 MiB and table bloat estimate >= 40.00% and < 90.00%:
- project_mirror_data: size 560.25 MiB, can be reduced 2.23 times, by ~308.94 MiB (~55.14%)
- lists: size 383.85 MiB, can be reduced 1.79 times, by ~168.83 MiB (~43.98%)
- clusters: size 5.17 MiB, can be reduced 1.94 times, by ~2.51 MiB (~48.56%)
- cluster_providers_gcp: size 3.23 MiB, can be reduced 2.03 times, by ~1.65 MiB (~50.85%)
- lfs_file_locks: size 1.82 MiB, can be reduced 5.66 times, by ~1.50 MiB (~82.33%)
- etc.

Recommendations

[P1] Reduce and prevent the high level of table bloat:
- to prevent a high level of bloat in the future, tune autovacuum: consider more aggressive autovacuum settings (see F001);
- eliminate or reduce the current table bloat using one of the approaches listed below.
If you want to get exact bloat numbers, clone the database, get table sizes, then apply database-wide VACUUM FULL (it eliminate all the bloat), and get new table sizes. Then compare old and new numbers.
To reduce the table bloat, consider one of the following approaches:
- VACUUM FULL (⚠ requires downtime / maintenance window),
- one of the tools reducing the bloat online, without interrupting the operations:
  - pg_repack,
  - pg_squeeze,
  - pgcompacttable.
Read more on this topic:
- Bloat estimation for tables (2014, ioguix)
- Show database bloat (PostgreSQL wiki)
- PostgreSQL Bloat: origins, monitoring and managing (2016, Compose)
- Dealing with significant Postgres database bloat — what are your options? (2018, Compass)
- Postgres database bloat analysis (2019, GitLab)

Table of contents

F005 Autovacuum: Btree Index Bloat (Estimated)

⚠ This report is based on estimations. The errors in bloat estimates may be significant (in some cases, up to 15% and even more). Use it only as an indicator of potential issues.

Observations

Data collected: 2019-11-11 13:40:17 +0000 UTC
Current database: gitlabhq_production
ℹ This report considers only btree indexes.

Master (`10.220.16.106`)

The list is limited to 20 items. Total: 983.

#	Index (Table)	Index Size	Table Size	▼ Estimated bloat	Est. bloat, bytes	Est. bloat factor	Est. bloat level, %	Live Data Size	Fillfactor
	===== TOTAL =====	1.30 TiB	6.09 TiB	373.47 GiB	401,000,194,048	1.39	28.21	0.93 TiB
1	`index_ci_builds_on_auto_canceled_by_id` (`ci_builds`*)	19.15 GiB	327.80 GiB	15.05 GiB	16,152,223,744	4.66	78.56	~4.11 GiB	90
2	`index_merge_request_diff_commits_on_sha` (`merge_request_diff_commits`)	94.31 GiB	523.79 GiB	14.57 GiB	15,643,197,440	1.18	15.45	~79.74 GiB	90
3	`index_ci_builds_on_token` (`ci_builds`*)	20.73 GiB	327.80 GiB	14.07 GiB	15,097,405,440	3.11	67.85	~6.67 GiB	90
4	`index_ci_builds_on_protected` (`ci_builds`*)	18.94 GiB	327.80 GiB	12.16 GiB	13,055,688,704	2.80	64.23	~6.78 GiB	90
6	`tmp_build_stage_position_index` (`ci_builds`*)	18.89 GiB	327.80 GiB	12.12 GiB	13,009,723,392	2.79	64.16	~6.77 GiB	90
7	`index_ci_builds_on_runner_id` (`ci_builds`*)	17.73 GiB	327.80 GiB	11.03 GiB	11,842,748,416	2.65	62.21	~6.70 GiB	90
8	`index_ci_builds_on_queued_at` (`ci_builds`*)	16.56 GiB	327.80 GiB	9.79 GiB	10,505,617,408	2.44	59.10	~6.78 GiB	90
9	`index_ci_builds_on_commit_id_and_type_and_ref` (`ci_builds`*)	43.15 GiB	327.80 GiB	9.01 GiB	9,665,478,656	1.26	20.87	~34.14 GiB	90
10	`index_ci_job_artifacts_on_expire_at_and_job_id` (`ci_job_artifacts`)	14.62 GiB	41.50 GiB	8.75 GiB	9,395,191,808	2.49	59.88	~5.87 GiB	90
11	`index_ci_builds_on_updated_at` (`ci_builds`*)	15.46 GiB	327.80 GiB	8.65 GiB	9,280,675,840	2.27	55.94	~6.81 GiB	90
12	`index_ci_builds_on_token_encrypted` (`ci_builds`*)	17.19 GiB	327.80 GiB	8.28 GiB	8,884,101,120	1.93	48.13	~8.92 GiB	90
13	`ci_builds_pkey` (`ci_builds`*)	14.22 GiB	327.80 GiB	7.41 GiB	7,951,065,088	2.09	52.10	~6.81 GiB	90
14	`index_ci_build_trace_sections_on_section_name_id` (`ci_build_trace_sections`)	37.98 GiB	108.64 GiB	6.26 GiB	6,721,126,400	1.20	16.49	~31.72 GiB	90
15	`index_ci_builds_on_stage_id` (`ci_builds`*)	12.64 GiB	327.80 GiB	5.83 GiB	6,255,837,184	1.86	46.11	~6.81 GiB	90
16	`index_ci_build_trace_sections_on_project_id` (`ci_build_trace_sections`)	37.16 GiB	108.64 GiB	5.45 GiB	5,847,441,408	1.17	14.66	~31.72 GiB	90
17	`index_projects_on_repository_storage` (`projects`*)	5.75 GiB	2.95 GiB	5.43 GiB	5,823,053,824	17.80	94.38	~330.58 MiB	90
18	`index_ci_builds_on_project_id_and_id` (`ci_builds`*)	14.88 GiB	327.80 GiB	5.37 GiB	5,763,760,128	1.56	36.08	~9.52 GiB	90
19	`index_ci_builds_on_user_id` (`ci_builds`*)	11.91 GiB	327.80 GiB	5.12 GiB	5,491,097,600	1.75	42.96	~6.80 GiB	90
20	`index_projects_on_last_activity_at` (`projects`*)	4.91 GiB	2.95 GiB	4.68 GiB	5,019,623,424	21.23	95.29	~236.65 MiB	90
21	`index_projects_on_runners_token` (`projects`*)	4.77 GiB	2.95 GiB	4.63 GiB	4,962,598,912	33.35	97.00	~146.31 MiB	90

* This table has specific autovacuum settings. See 'F001 Autovacuum: Current settings'

Conclusions

[P1] Total index (btree only) bloat estimation is ~373.47 GiB, it is 28.21% of the overall size of all indexes and 8.29% of the DB size. Removing the index bloat will reduce the total DB size down to ~4.04 TiB. Free disk space will be increased by ~373.47 GiB. Total size of indexes is 1.39 times bigger than it could be. Notice that this is only an estimation, sometimes it may be significantly off.
[P1] The following 41 indexes have significant size (>1 MiB) and bloat estimate > 90.00%:
- index_projects_on_repository_storage: size 5.75 GiB, can be reduced 17.80 times, by ~5.46 GiB (~94.38%)
- index_projects_on_last_activity_at: size 4.91 GiB, can be reduced 21.23 times, by ~4.70 GiB (~95.29%)
- index_projects_on_runners_token: size 4.77 GiB, can be reduced 33.35 times, by ~4.64 GiB (~97.00%)
- index_projects_on_pending_delete: size 4.78 GiB, can be reduced 20.65 times, by ~4.57 GiB (~95.16%)
- index_projects_on_last_repository_check_failed: size 4.65 GiB, can be reduced 33.56 times, by ~4.53 GiB (~97.02%)
- etc.
[P2] There are 200 indexes with size > 1 MiB and index bloat estimate >= 40.00% and < 90.00%:
- index_ci_builds_on_auto_canceled_by_id: size 19.15 GiB, can be reduced 4.66 times, by ~15.46 GiB (~78.56%)
- index_ci_builds_on_token: size 20.73 GiB, can be reduced 3.11 times, by ~14.72 GiB (~67.85%)
- index_ci_builds_on_protected: size 18.94 GiB, can be reduced 2.80 times, by ~12.84 GiB (~64.23%)
- tmp_build_stage_position_index: size 18.89 GiB, can be reduced 2.79 times, by ~12.79 GiB (~64.16%)
- index_ci_builds_on_runner_id: size 17.73 GiB, can be reduced 2.65 times, by ~11.70 GiB (~62.21%)
- etc.

Recommendations

[P1] Reduce and prevent a high level of index bloat:
- to prevent a high level of bloat in the future, tune autovacuum: consider more aggressive autovacuum settings (see F001);
- eliminate or reduce the current index bloat using one of the approaches listed below.
If you want to get exact bloat numbers, clone the database, get index sizes, then apply database-wide VACUUM FULL (it eliminates all the bloat), and gets new table sizes. Then compare old and new numbers.
To reduce the index bloat, consider one of the following approaches:
- VACUUM FULL (⚠ requires downtime / maintenance window),
- REINDEX (REINDEX INDEX, REINDEX TABLE; ⚠ requires downtime / maintenance window),
- recreating indexes online using CREATE INDEX CONCURRENTLY, DROP INDEX CONCURRENTLY and renaming (not trivial for indexes supporting PK, FK) // REINDEX CONCURRENTLY is available in Postgres 12+,
- one of the tools reducing the bloat online, without interrupting the operations:
  - pg_repack,
  - pg_squeeze,
  - pgcompacttable.
Read more on this topic:
- Index maintenance (PostgreSQL wiki)
- Btree bloat query (2014, ioguix)
- PostgreSQL Index bloat under a microscope (2017, Peter Geoghegan)
- PostgreSQL Bloat: origins, monitoring and managing (2016, Compose)
- Dealing with significant Postgres database bloat — what are your options? (2018, Compass)
- Postgres database bloat analysis (2019, GitLab)

Table of contents

F008 Autovacuum: Resource Usage

Observations

Data collected: 2019-11-11 13:41:05 +0000 UTC

Settings

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_vacuum_cost_delay	5	ms
autovacuum_vacuum_cost_limit	6000
autovacuum_work_mem	-1	kB
log_autovacuum_min_duration	0	ms
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
vacuum_cost_limit	200
work_mem	102400	kB	100.00 MiB

CPU

Cpu count you can see in report A001

RAM

Ram amount you can see in report A001

Max workers memory: 12 GiB

DISK

⚠ Warning: collection of current impact on disks is not yet implemented. Please refer to Postgres logs and see current read and write IO bandwidth caused by autovacuum.

Conclusions

Recommendations

Table of contents

G001 Memory-related Settings

Observations

Data collected: 2019-11-11 13:47:52 +0000 UTC

Master (`10.220.16.106`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica servers:

Replica (`10.220.16.101`)

Nothing found

Replica (`10.220.16.102`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica (`10.220.16.103`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica (`10.220.16.104`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica (`10.220.16.105`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica (`10.220.16.107`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica (`10.220.16.108`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Replica (`10.220.16.109`)

Setting name	Value	Unit	Pretty value
autovacuum_max_workers	6
autovacuum_work_mem	-1	kB
effective_cache_size	43352064	8kB	330.75 GiB
maintenance_work_mem	2097152	kB	2.00 GiB
max_connections	300
shared_buffers	14450688	8kB	110.25 GiB
temp_buffers	1024	8kB	8.00 MiB
work_mem	102400	kB	100.00 MiB

Conclusions

Recommendations

Table of contents

G002 Connections and Current Activity

Observations

Data collected: 2019-11-11 13:47:53 +0000 UTC

Master (`10.220.16.106`)

The list is limited to 20 items. Total: 19.

#	User	DB	Current state	Count	State changed >1m ago	Tx age >1m
1	ALL users	ALL databases	ALL states	261	1	1
2	`gitlab`	ALL databases	active	14	0	0
3	`gitlab-psql`	ALL databases	active	2	1	1
4	`gitlab-checkup`	ALL databases	active	1	0	0
5	`gitlab`	ALL databases	idle	180	0	0
6	`pgbouncer`	ALL databases	idle	4	0	0
7	`gitlab-monitor`	ALL databases	idle	2	0	0
8	`gitlab-superuser`	ALL databases	idle	1	0	0
9	`postgres_exporter`	ALL databases	idle	1	0	0
10	`gitlab`	ALL databases	idle in transaction	56	0	0
11	`gitlab`	`gitlabhq_production`	active	14	0	0
12	`gitlab-psql`	`gitlabhq_production`	active	2	1	1
13	`gitlab-checkup`	`gitlabhq_production`	active	1	0	0
14	`gitlab`	`gitlabhq_production`	idle	180	0	0
15	`pgbouncer`	`gitlabhq_production`	idle	4	0	0
16	`gitlab-monitor`	`gitlabhq_production`	idle	2	0	0
17	`postgres_exporter`	`gitlabhq_production`	idle	1	0	0
18	`gitlab`	`gitlabhq_production`	idle in transaction	56	0	0
19	`gitlab-superuser`	`postgres`	idle	1	0	0

Replica servers:

Replica (`10.220.16.101`)

Nothing found

Replica (`10.220.16.102`)

#	User	DB	Current state	Count	State changed >1m ago
1	ALL users	ALL databases	ALL states	39	2
2	`gitlab`	ALL databases	active	2	0
3	`gitlab-monitor`	ALL databases	active	1	0
4	`gitlab-checkup`	ALL databases	active	1	0
5	`gitlab`	ALL databases	idle	29	1
6	`pgbouncer`	ALL databases	idle	4	1
7	`gitlab-superuser`	ALL databases	idle	1	0
8	`postgres_exporter`	ALL databases	idle	1	0
9	`gitlab`	`gitlabhq_production`	active	2	0
10	`gitlab-checkup`	`gitlabhq_production`	active	1	0
11	`gitlab-monitor`	`gitlabhq_production`	active	1	0
12	`gitlab`	`gitlabhq_production`	idle	29	1
13	`pgbouncer`	`gitlabhq_production`	idle	4	1
14	`postgres_exporter`	`gitlabhq_production`	idle	1	0
15	`gitlab-superuser`	`postgres`	idle	1	0

Replica (`10.220.16.103`)

#	User	DB	Current state	Count	State changed >1m ago
1	ALL users	ALL databases	ALL states	39	1
2	`gitlab`	ALL databases	active	6	0
3	`gitlab-checkup`	ALL databases	active	1	0
4	`gitlab`	ALL databases	idle	26	1
5	`pgbouncer`	ALL databases	idle	3	0
6	`gitlab-monitor`	ALL databases	idle	1	0
7	`gitlab-superuser`	ALL databases	idle	1	0
8	`postgres_exporter`	ALL databases	idle	1	0
9	`gitlab`	`gitlabhq_production`	active	6	0
10	`gitlab-checkup`	`gitlabhq_production`	active	1	0
11	`gitlab`	`gitlabhq_production`	idle	26	1
12	`pgbouncer`	`gitlabhq_production`	idle	3	0
13	`postgres_exporter`	`gitlabhq_production`	idle	1	0
14	`gitlab-monitor`	`gitlabhq_production`	idle	1	0
15	`gitlab-superuser`	`postgres`	idle	1	0

Replica (`10.220.16.104`)

#	User	DB	Current state	Count
1	ALL users	ALL databases	ALL states	41
2	`gitlab`	ALL databases	active	9
3	`gitlab-checkup`	ALL databases	active	1
4	`gitlab`	ALL databases	idle	26
5	`pgbouncer`	ALL databases	idle	2
6	`gitlab-monitor`	ALL databases	idle	1
7	`gitlab-superuser`	ALL databases	idle	1
8	`postgres_exporter`	ALL databases	idle	1
9	`gitlab`	`gitlabhq_production`	active	9
10	`gitlab-checkup`	`gitlabhq_production`	active	1
11	`gitlab`	`gitlabhq_production`	idle	26
12	`pgbouncer`	`gitlabhq_production`	idle	2
13	`postgres_exporter`	`gitlabhq_production`	idle	1
14	`gitlab-monitor`	`gitlabhq_production`	idle	1
15	`gitlab-superuser`	`postgres`	idle	1

Replica (`10.220.16.105`)

#	User	DB	Current state	Count
1	ALL users	ALL databases	ALL states	38
2	`gitlab`	ALL databases	active	7
3	`gitlab-checkup`	ALL databases	active	1
4	`gitlab`	ALL databases	idle	25
5	`pgbouncer`	ALL databases	idle	2
6	`gitlab-monitor`	ALL databases	idle	1
7	`gitlab-superuser`	ALL databases	idle	1
8	`postgres_exporter`	ALL databases	idle	1
9	`gitlab`	`gitlabhq_production`	active	7
10	`gitlab-checkup`	`gitlabhq_production`	active	1
11	`gitlab`	`gitlabhq_production`	idle	25
12	`pgbouncer`	`gitlabhq_production`	idle	2
13	`postgres_exporter`	`gitlabhq_production`	idle	1
14	`gitlab-monitor`	`gitlabhq_production`	idle	1
15	`gitlab-superuser`	`postgres`	idle	1

Replica (`10.220.16.107`)

#	User	DB	Current state	Count
1	ALL users	ALL databases	ALL states	40
2	`gitlab`	ALL databases	active	5
3	`gitlab-checkup`	ALL databases	active	1
4	`gitlab`	ALL databases	idle	28
5	`pgbouncer`	ALL databases	idle	3
6	`gitlab-monitor`	ALL databases	idle	1
7	`gitlab-superuser`	ALL databases	idle	1
8	`postgres_exporter`	ALL databases	idle	1
9	`gitlab`	`gitlabhq_production`	active	5
10	`gitlab-checkup`	`gitlabhq_production`	active	1
11	`gitlab`	`gitlabhq_production`	idle	28
12	`pgbouncer`	`gitlabhq_production`	idle	3
13	`postgres_exporter`	`gitlabhq_production`	idle	1
14	`gitlab-monitor`	`gitlabhq_production`	idle	1
15	`gitlab-superuser`	`postgres`	idle	1

Replica (`10.220.16.108`)

#	User	DB	Current state	Count
1	ALL users	ALL databases	ALL states	44
2	`gitlab`	ALL databases	active	3
3	`gitlab-checkup`	ALL databases	active	1
4	`gitlab`	ALL databases	idle	34
5	`pgbouncer`	ALL databases	idle	3
6	`gitlab-monitor`	ALL databases	idle	1
7	`gitlab-superuser`	ALL databases	idle	1
8	`postgres_exporter`	ALL databases	idle	1
9	`gitlab`	`gitlabhq_production`	active	3
10	`gitlab-checkup`	`gitlabhq_production`	active	1
11	`gitlab`	`gitlabhq_production`	idle	34
12	`pgbouncer`	`gitlabhq_production`	idle	3
13	`postgres_exporter`	`gitlabhq_production`	idle	1
14	`gitlab-monitor`	`gitlabhq_production`	idle	1
15	`gitlab-superuser`	`postgres`	idle	1

Replica (`10.220.16.109`)

#	User	DB	Current state	Count
1	ALL users	ALL databases	ALL states	40
2	`gitlab`	ALL databases	active	8
3	`gitlab-checkup`	ALL databases	active	1
4	`gitlab`	ALL databases	idle	25
5	`pgbouncer`	ALL databases	idle	3
6	`gitlab-monitor`	ALL databases	idle	1
7	`gitlab-superuser`	ALL databases	idle	1
8	`postgres_exporter`	ALL databases	idle	1
9	`gitlab`	`gitlabhq_production`	active	8
10	`gitlab-checkup`	`gitlabhq_production`	active	1
11	`gitlab`	`gitlabhq_production`	idle	25
12	`pgbouncer`	`gitlabhq_production`	idle	3
13	`postgres_exporter`	`gitlabhq_production`	idle	1
14	`gitlab-monitor`	`gitlabhq_production`	idle	1
15	`gitlab-superuser`	`postgres`	idle	1

Conclusions

Recommendations

All good, no recommendations here.

Table of contents

G003 Timeouts, Locks, Deadlocks

Observations

Data collected: 2019-11-11 13:47:54 +0000 UTC

Master (`10.220.16.106`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

#	Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
1	`gitlabhq_production`	4	217	2019-10-28T16:07:34.673503+00:00	13 days 21:33:32
2	`postgres`	0	0	2019-10-28T16:07:29.861973+00:00	13 days 21:33:37

Replica servers:

Replica (`10.220.16.101`)

Replica (`10.220.16.102`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T09:21:58.903529+00:00	13 days 04:13:58
`gitlabhq_production`	0	0	2019-10-29T09:21:58.736373+00:00	13 days 04:13:59

Replica (`10.220.16.103`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T09:21:59.465545+00:00	13 days 04:14:40
`gitlabhq_production`	0	0	2019-10-29T09:21:59.999464+00:00	13 days 04:14:39

Replica (`10.220.16.104`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T09:21:59.182724+00:00	13 days 04:15:34
`gitlabhq_production`	0	0	2019-10-29T09:21:59.913905+00:00	13 days 04:15:34

Replica (`10.220.16.105`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T09:21:58.542843+00:00	13 days 04:16:45
`gitlabhq_production`	0	0	2019-10-29T09:21:58.540354+00:00	13 days 04:16:45

Replica (`10.220.16.107`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T17:48:31.100965+00:00	12 days 19:54:29
`gitlabhq_production`	0	0	2019-10-29T17:48:33.958403+00:00	12 days 19:54:26

Replica (`10.220.16.108`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T09:22:01.644637+00:00	13 days 04:23:14
`gitlabhq_production`	0	0	2019-10-29T09:22:05.461979+00:00	13 days 04:23:10

Replica (`10.220.16.109`)

Timeouts

Setting name	Value	Unit
authentication_timeout	60	s
idle_in_transaction_session_timeout	30000	ms
statement_timeout	15000	ms

Locks

Setting name	Value	Unit
deadlock_timeout	5000	ms
lock_timeout	0	ms
max_locks_per_transaction	128
max_pred_locks_per_transaction	64

Databases data

Database	Conflicts	▼ Deadlocks	Stats reset at	Stat reset
`postgres`	0	0	2019-10-29T09:22:04.730022+00:00	13 days 04:25:50
`gitlabhq_production`	0	0	2019-10-29T09:22:06.133796+00:00	13 days 04:25:48

Conclusions

Recommendations

Table of contents

H001 Invalid Indexes

Observations

Data collected: 2019-11-11 13:41:07 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Invalid indexes not found

Conclusions

Recommendations

Table of contents

H002 Unused and Redundant Indexes

Observations

Data collected: 2019-11-11 13:47:54 +0000 UTC
Current database: gitlabhq_production
Stats reset: 13 days 21:34:00 ago (2019-10-28 16:07:35 +0000 UTC)
⚠ Statistics age is less than 30 days. Make decisions on index cleanup with caution!

Never Used Indexes

The list is limited to 20 items. Total: 68.

#	Table	Index	10.220.16.106 usage	10.220.16.102 usage	10.220.16.103 usage	10.220.16.104 usage	10.220.16.105 usage	10.220.16.107 usage	10.220.16.108 usage	10.220.16.109 usage	▼ Index size	Table size	Supports FK
	=====TOTAL=====										285.58 GiB	2.26 TiB
1	`ci_builds`	`index_ci_builds_on_protected`	0	0	0	0	0	0	0	0	18.97 GiB	327.71 GiB
2	`ci_builds`	`index_ci_builds_on_queued_at`	0	0		0	0	0	0	0	16.60 GiB	327.71 GiB
3	`ci_job_artifacts`	`index_ci_job_artifacts_on_file_store`	0	0	0	0	0	0	0	0	7.73 GiB	41.49 GiB
4	`audit_events`	`analytics_index_audit_events_on_created_at_and_author_id`	0	0	0	0	0	0	0	0	5.95 GiB	46.63 GiB
7	`projects`	`index_projects_on_pending_delete`	0	0	0	0	0	0	0	0	4.78 GiB	2.95 GiB	Yes
17	`projects`	`index_projects_on_runners_token`	0	0	0	0	0	0	0	0	4.77 GiB	2.95 GiB	Yes
20	`projects`	`index_projects_on_last_repository_check_failed`	0	0	0	0	0	0	0	0	4.65 GiB	2.95 GiB	Yes
29	`projects`	`index_projects_on_mirror_last_successful_update_at`	0	0	0	0	0	0	0	0	4.64 GiB	2.95 GiB	Yes
30	`merge_requests`	`index_merge_requests_on_title`	0								2.97 GiB	14.77 GiB
34	`issues`	`index_issues_on_project_id_and_updated_at_and_id_and_state`	0	0	0	0	0	0	0	0	2.04 GiB	13.40 GiB	Yes
64	`users`	`index_users_on_accepted_term_id`	0	0	0	0	0	0	0	0	2.04 GiB	2.06 GiB	Yes
66	`issues`	`index_issues_on_state`	0	0	0	0	0	0	0	0	1.94 GiB	13.40 GiB	Yes
70	`issues`	`index_issues_on_project_id_and_created_at_and_id_and_state`	0	0	0	0	0	0	0	0	1.86 GiB	13.40 GiB	Yes
71	`merge_request_diffs`	`index_merge_request_diffs_on_merge_request_id_and_id_partial`	0	0	0	0	0	0	0	0	1.47 GiB	11.66 GiB	Yes
72	`projects`	`idx_project_repository_check_partial`	0	0	0	0	0	0	0	0	1.38 GiB	2.95 GiB
73	`merge_request_metrics`	`index_merge_request_metrics_on_first_deployed_to_production_at`	0	0	0	0	0	0	0	0	1.36 GiB	2.30 GiB
74	`ci_runners`	`index_ci_runners_on_is_shared`	0	0	0	0	0	0	0	0	922.22 MiB	107.80 MiB
75	`ci_runners`	`index_ci_runners_on_locked`	0	0	0	0	0	0	0	0	885.41 MiB	107.80 MiB
91	`users`	`index_users_on_state_and_internal`	0	0	0	0	0	0	0	0	876.11 MiB	2.06 GiB	Yes
92	`ci_runners`	`index_ci_runners_on_contacted_at`	0	0			0		0	0	797.01 MiB	107.80 MiB

Rarely Used Indexes

The list is limited to 20 items. Total: 228.

#	Table	Index	10.220.16.106 usage	10.220.16.102 usage	10.220.16.103 usage	10.220.16.104 usage	10.220.16.105 usage	10.220.16.107 usage	10.220.16.108 usage	10.220.16.109 usage	▼ Index size	Table size	Comment	Supports FK
	=====TOTAL=====										1.60 TiB	11.15 TiB
1	`merge_request_diff_commits`	`index_merge_request_diff_commits_on_sha`	scans: 10,553,058/hour, writes: 74,459,249/hour	scans: 81,958/hour, writes: 0/hour	scans: 82,108/hour, writes: 0/hour	scans: 82,018/hour, writes: 0/hour	scans: 82,136/hour, writes: 0/hour	scans: 78,267/hour, writes: 0/hour	scans: 82,224/hour, writes: 0/hour	scans: 82,175/hour, writes: 0/hour	94.38 GiB	523.64 GiB	Low Scans, High Writes
2	`ci_build_trace_sections`	`index_ci_build_trace_sections_on_section_name_id`	scans: 69,308/hour, writes: 71,099,131/hour								38.02 GiB	108.61 GiB	Low Scans, High Writes
3	`ci_build_trace_sections`	`index_ci_build_trace_sections_on_project_id`	scans: 65,170/hour, writes: 71,099,131/hour								37.20 GiB	108.61 GiB	Low Scans, High Writes
5	`ci_builds`	`index_ci_builds_on_auto_canceled_by_id`	scans: 222,125/hour, writes: 44,324,980/hour								19.19 GiB	327.71 GiB	Low Scans, High Writes	Yes
6	`ci_builds`	`tmp_build_stage_position_index`	scans: 10/hour, writes: 44,324,980/hour								18.92 GiB	327.71 GiB	Low Scans, High Writes
7	`events`	`index_events_on_project_id_and_created_at`	scans: 11,107/hour, writes: 13,150,910/hour	scans: 170,936/hour, writes: 0/hour	scans: 156,953/hour, writes: 0/hour	scans: 161,076/hour, writes: 0/hour	scans: 164,277/hour, writes: 0/hour	scans: 119,837/hour, writes: 0/hour	scans: 153,201/hour, writes: 0/hour	scans: 156,391/hour, writes: 0/hour	18.11 GiB	31.31 GiB	Low Scans, High Writes
9	`ci_builds`	`index_ci_builds_on_runner_id`	scans: 6,380,876/hour, writes: 44,324,980/hour	scans: 1,273,422/hour, writes: 0/hour	scans: 1,272,480/hour, writes: 0/hour	scans: 1,272,009/hour, writes: 0/hour	scans: 1,274,865/hour, writes: 0/hour	scans: 1,256,957/hour, writes: 0/hour	scans: 1,270,666/hour, writes: 0/hour	scans: 1,277,494/hour, writes: 0/hour	17.77 GiB	327.71 GiB	Low Scans, High Writes	Yes
10	`ci_builds`	`index_ci_builds_on_status_and_type_and_runner_id`	scans: 26,798,375/hour, writes: 44,324,980/hour	scans: 4,698,630/hour, writes: 0/hour	scans: 4,698,702/hour, writes: 0/hour	scans: 4,687,021/hour, writes: 0/hour	scans: 4,697,975/hour, writes: 0/hour	scans: 4,627,686/hour, writes: 0/hour	scans: 4,683,278/hour, writes: 0/hour	scans: 4,712,925/hour, writes: 0/hour	16.95 GiB	327.71 GiB	Low Scans, High Writes
11	`notes`	`index_notes_on_discussion_id`	scans: 4,883,156/hour, writes: 9,453,665/hour	scans: 68,047/hour, writes: 0/hour	scans: 67,231/hour, writes: 0/hour	scans: 67,820/hour, writes: 0/hour	scans: 68,453/hour, writes: 0/hour	scans: 64,936/hour, writes: 0/hour	scans: 68,706/hour, writes: 0/hour	scans: 68,546/hour, writes: 0/hour	15.61 GiB	133.66 GiB	Low Scans, High Writes
12	`ci_builds`	`index_ci_builds_on_updated_at`	scans: 1,684/hour, writes: 44,324,980/hour		scans: 27/hour, writes: 0/hour		scans: 5/hour, writes: 0/hour				15.49 GiB	327.71 GiB	Low Scans, High Writes
13	`ci_job_artifacts`	`index_ci_job_artifacts_on_expire_at_and_job_id`	scans: 28,890/hour, writes: 29,538,977/hour								14.63 GiB	41.49 GiB	Low Scans, High Writes
14	`events`	`analytics_index_events_on_created_at_and_author_id`	scans: 36,913/hour, writes: 13,150,910/hour	scans: 344,359/hour, writes: 0/hour	scans: 344,229/hour, writes: 0/hour	scans: 345,001/hour, writes: 0/hour	scans: 345,093/hour, writes: 0/hour	scans: 333,433/hour, writes: 0/hour	scans: 345,531/hour, writes: 0/hour	scans: 344,961/hour, writes: 0/hour	14.16 GiB	31.31 GiB	Low Scans, High Writes
15	`ci_builds`	`ci_builds_gitlab_monitor_metrics`	scans: 2,766/hour, writes: 44,324,980/hour	scans: 3,423/hour, writes: 0/hour	scans: 3,392/hour, writes: 0/hour	scans: 3,551/hour, writes: 0/hour	scans: 3,553/hour, writes: 0/hour	scans: 3,472/hour, writes: 0/hour	scans: 3,430/hour, writes: 0/hour	scans: 3,500/hour, writes: 0/hour	13.66 GiB	327.71 GiB	Low Scans, High Writes
17	`ci_builds`	`index_ci_builds_on_user_id`	scans: 2,010/hour, writes: 44,324,980/hour								11.93 GiB	327.71 GiB	Low Scans, High Writes	Yes
18	`events`	`index_events_on_project_id_and_id`	scans: 2,625,661/hour, writes: 13,150,910/hour								11.52 GiB	31.31 GiB	Low Scans, High Writes
19	`events`	`index_events_on_action`	scans: 10/hour, writes: 13,150,910/hour	scans: 120/hour, writes: 0/hour	scans: 152/hour, writes: 0/hour	scans: 150/hour, writes: 0/hour	scans: 158/hour, writes: 0/hour	scans: 160/hour, writes: 0/hour	scans: 118/hour, writes: 0/hour	scans: 108/hour, writes: 0/hour	11.29 GiB	31.31 GiB	Low Scans, High Writes
20	`ci_job_artifacts`	`index_ci_job_artifacts_on_project_id`	scans: 68,649/hour, writes: 29,538,977/hour	scans: 596/hour, writes: 0/hour	scans: 589/hour, writes: 0/hour	scans: 602/hour, writes: 0/hour	scans: 545/hour, writes: 0/hour		scans: 579/hour, writes: 0/hour	scans: 547/hour, writes: 0/hour	9.55 GiB	41.49 GiB	Low Scans, High Writes
21	`notes`	`index_notes_on_line_code`	scans: 7,781/hour, writes: 9,453,665/hour				scans: 2/hour, writes: 0/hour				7.56 GiB	133.66 GiB	Low Scans, High Writes
22	`ci_builds_metadata`	`index_ci_builds_metadata_on_project_id`	scans: 65,169/hour, writes: 19,530,031/hour								6.79 GiB	12.82 GiB	Low Scans, High Writes
23	`notes`	`index_notes_on_review_id`	scans: 87,225/hour, writes: 9,453,665/hour								6.25 GiB	133.66 GiB	Low Scans, High Writes

Redundant Indexes

#	Table	Index	Redundant to	10.220.16.106 usage	▼ Index size	Table size
	=====TOTAL=====				25.09 MiB	69.52 MiB
1	`namespace_root_storage_statistics`	`index_namespace_root_storage_statistics_on_namespace_id`	`namespace_root_storage_statistics_pkey`	0	19.18 MiB	61.51 MiB
2	`user_statuses`	`index_user_statuses_on_user_id`	`user_statuses_pkey`	0	2.39 MiB	5.97 MiB
3	`namespace_aggregation_schedules`	`index_namespace_aggregation_schedules_on_namespace_id`	`namespace_aggregation_schedules_pkey`	0	2.36 MiB	712.00 KiB
4	`project_feature_usages`	`index_project_feature_usages_on_project_id`	`project_feature_usages_pkey`	0	1.18 MiB	1.35 MiB
5	`aws_roles`	`index_aws_roles_on_user_id`	`aws_roles_pkey`	0	8.00 KiB	0.00 bytes

Conclusions

Recommendations

"DO" database migration code

DROP INDEX CONCURRENTLY index_analytics_ca_group_stages_on_group_id; -- 8192 bytes, index_analytics_ca_group_stages_on_group_id_and_name, table analytics_cycle_analytics_group_stages
DROP INDEX CONCURRENTLY index_analytics_ca_project_stages_on_project_id; -- 8192 bytes, index_analytics_ca_project_stages_on_project_id_and_name, table analytics_cycle_analytics_project_stages
DROP INDEX CONCURRENTLY index_approval_merge_request_rules_1; -- 144 MB, approval_rule_name_index_for_code_owners, table approval_merge_request_rules
DROP INDEX CONCURRENTLY index_approval_rules_code_owners_rule_type; -- 3832 kB, index_approval_rule_name_for_code_owners_rule_type, any_approver_merge_request_rule_type_unique_index, table approval_merge_request_rules
DROP INDEX CONCURRENTLY index_approval_project_rules_on_project_id; -- 2768 kB, any_approver_project_rule_type_unique_index, table approval_project_rules
DROP INDEX CONCURRENTLY analytics_index_audit_events_on_created_at_and_author_id; -- 6091 MB, Never Used Indexes, table audit_events
DROP INDEX CONCURRENTLY index_aws_roles_on_user_id; -- 8192 bytes, aws_roles_pkey, table aws_roles
DROP INDEX CONCURRENTLY index_board_group_recent_visits_on_user_id; -- 744 kB, index_board_group_recent_visits_on_user_group_and_board, table board_group_recent_visits
DROP INDEX CONCURRENTLY index_board_project_recent_visits_on_user_id; -- 2632 kB, index_board_project_recent_visits_on_user_project_and_board, table board_project_recent_visits
DROP INDEX CONCURRENTLY index_ci_builds_on_artifacts_expire_at; -- 82 MB, Never Used Indexes, table ci_builds
DROP INDEX CONCURRENTLY index_ci_builds_on_project_id_for_successfull_pages_deploy; -- 73 MB, index_ci_builds_project_id_and_status_for_live_jobs_partial2, Never Used Indexes, Never Used Indexes, index_ci_builds_on_project_id_and_id, table ci_builds
DROP INDEX CONCURRENTLY index_ci_builds_on_protected; -- 19 GB, Never Used Indexes, table ci_builds
DROP INDEX CONCURRENTLY index_ci_builds_on_queued_at; -- 17 GB, Never Used Indexes, table ci_builds
DROP INDEX CONCURRENTLY index_ci_builds_on_stage_id; -- 13 GB, tmp_build_stage_position_index, table ci_builds
DROP INDEX CONCURRENTLY index_ci_builds_metadata_on_build_id_and_has_exposed_artifacts; -- 40 kB, Never Used Indexes, index_ci_builds_metadata_on_build_id, index_ci_builds_metadata_on_build_id_and_interruptible, table ci_builds_metadata
DROP INDEX CONCURRENTLY index_ci_builds_metadata_on_build_id_and_interruptible; -- 3208 kB, index_ci_builds_metadata_on_build_id_and_has_exposed_artifacts, index_ci_builds_metadata_on_build_id, table ci_builds_metadata
DROP INDEX CONCURRENTLY index_ci_job_artifacts_on_file_store; -- 7909 MB, Never Used Indexes, table ci_job_artifacts
DROP INDEX CONCURRENTLY index_ci_job_artifacts_on_project_id; -- 9770 MB, index_ci_job_artifacts_on_project_id_for_security_reports, table ci_job_artifacts
DROP INDEX CONCURRENTLY index_ci_job_artifacts_on_project_id_for_security_reports; -- 21 MB, index_ci_job_artifacts_on_project_id, table ci_job_artifacts
DROP INDEX CONCURRENTLY index_ci_pipeline_schedules_on_next_run_at_and_active; -- 25 MB, Never Used Indexes, table ci_pipeline_schedules
DROP INDEX CONCURRENTLY index_ci_pipelines_on_project_id; -- 2676 MB, index_ci_pipelines_on_project_id_and_source, index_ci_pipelines_on_project_id_and_ref_and_status_and_id, index_ci_pipelines_on_project_id_and_sha, index_ci_pipelines_on_project_id_and_status_and_config_source, index_ci_pipelines_on_project_idandrefandiddesc, index_ci_pipelines_on_project_id_and_iid, table ci_pipelines
DROP INDEX CONCURRENTLY index_ci_runners_on_contacted_at; -- 797 MB, Never Used Indexes, table ci_runners
DROP INDEX CONCURRENTLY index_ci_runners_on_is_shared; -- 922 MB, Never Used Indexes, table ci_runners
DROP INDEX CONCURRENTLY index_ci_runners_on_locked; -- 885 MB, Never Used Indexes, table ci_runners
DROP INDEX CONCURRENTLY index_ci_runners_on_token; -- 748 MB, Never Used Indexes, index_ci_runners_on_token_encrypted, table ci_runners
DROP INDEX CONCURRENTLY index_ci_stages_on_pipeline_id; -- 4057 MB, index_ci_stages_on_pipeline_id_and_name, index_ci_stages_on_pipeline_id_and_position, table ci_stages
DROP INDEX CONCURRENTLY index_cluster_providers_gcp_on_cloud_run; -- 256 kB, Never Used Indexes, table cluster_providers_gcp
DROP INDEX CONCURRENTLY index_clusters_kubernetes_namespaces_on_cluster_id; -- 1352 kB, kubernetes_namespaces_cluster_and_namespace, index_kubernetes_namespaces_on_cluster_project_environment_id, table clusters_kubernetes_namespaces
DROP INDEX CONCURRENTLY index_container_repositories_on_project_id; -- 17 MB, index_container_repositories_on_project_id_and_name, table container_repositories
DROP INDEX CONCURRENTLY index_deploy_tokens_on_token_and_expires_at_and_id; -- 12 MB, Never Used Indexes, table deploy_tokens
DROP INDEX CONCURRENTLY index_deployments_on_project_id_and_status; -- 1043 MB, index_deployments_on_project_id_and_status_and_created_at, table deployments
DROP INDEX CONCURRENTLY partial_index_deployments_for_legacy_successful_deployments; -- 16 kB, index_deployments_on_cluster_id_and_status, deployments_pkey, Never Used Indexes, table deployments
DROP INDEX CONCURRENTLY index_design_management_designs_versions_on_design_id; -- 72 kB, design_management_designs_versions_uniqueness, table design_management_designs_versions
DROP INDEX CONCURRENTLY index_design_management_designs_versions_on_event; -- 96 kB, Never Used Indexes, table design_management_designs_versions
DROP INDEX CONCURRENTLY index_epics_on_group_id; -- 920 kB, index_epics_on_start_date_sourcing_epic_id, index_epics_on_due_date_sourcing_epic_id, table epics
DROP INDEX CONCURRENTLY index_milestone; -- 1792 kB, index_epics_on_parent_id, index_epics_on_closed_by_id, table epics
DROP INDEX CONCURRENTLY index_geo_node_namespace_links_on_geo_node_id; -- 8192 bytes, index_geo_node_namespace_links_on_geo_node_id_and_namespace_id, table geo_node_namespace_links
DROP INDEX CONCURRENTLY index_issue_links_on_source_id; -- 9928 kB, index_issue_links_on_source_id_and_target_id, table issue_links
DROP INDEX CONCURRENTLY index_issue_metrics; -- 411 MB, index_issue_metrics_on_issue_id_and_timestamps, table issue_metrics
DROP INDEX CONCURRENTLY index_issues_on_project_id_and_created_at_and_id_and_state; -- 1903 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table issues
DROP INDEX CONCURRENTLY index_issues_on_project_id_and_updated_at_and_id_and_state; -- 2088 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table issues
DROP INDEX CONCURRENTLY index_issues_on_state; -- 1982 MB, Never Used Indexes, Never Used Indexes, table issues
DROP INDEX CONCURRENTLY idx_jira_connect_subscriptions_on_installation_id; -- 16 kB, Never Used Indexes, idx_jira_connect_subscriptions_on_installation_id_namespace_id, table jira_connect_subscriptions
DROP INDEX CONCURRENTLY index_label_priorities_on_priority; -- 3432 kB, Never Used Indexes, table label_priorities
DROP INDEX CONCURRENTLY index_list_user_preferences_on_user_id; -- 2576 kB, index_list_user_preferences_on_user_id_and_list_id, table list_user_preferences
DROP INDEX CONCURRENTLY index_members_on_access_level; -- 379 MB, Never Used Indexes, table members
DROP INDEX CONCURRENTLY index_merge_request_assignees_on_merge_request_id; -- 334 MB, index_merge_request_assignees_on_merge_request_id_and_user_id, table merge_request_assignees
DROP INDEX CONCURRENTLY index_merge_request_diffs_on_merge_request_id_and_id; -- 1406 MB, index_merge_request_diffs_on_merge_request_id_and_id_partial, table merge_request_diffs
DROP INDEX CONCURRENTLY index_merge_request_diffs_on_merge_request_id_and_id_partial; -- 1503 MB, Never Used Indexes, index_merge_request_diffs_on_merge_request_id_and_id, table merge_request_diffs
DROP INDEX CONCURRENTLY index_merge_request_metrics; -- 1000 MB, index_merge_request_metrics_on_merge_request_id_and_merged_at, table merge_request_metrics
DROP INDEX CONCURRENTLY index_merge_request_metrics_on_first_deployed_to_production_at; -- 1392 MB, Never Used Indexes, table merge_request_metrics
DROP INDEX CONCURRENTLY index_merge_request_metrics_on_latest_closed_at; -- 39 MB, Never Used Indexes, table merge_request_metrics
DROP INDEX CONCURRENTLY idx_merge_requests_on_id_and_merge_jid; -- 3376 kB, index_merge_requests_on_id_and_merge_jid, table merge_requests
DROP INDEX CONCURRENTLY idx_merge_requests_on_source_project_and_branch_state_opened; -- 113 MB, index_merge_requests_on_source_project_id_and_source_branch, index_merge_requests_on_source_project_and_branch_state_opened, table merge_requests
DROP INDEX CONCURRENTLY idx_merge_requests_on_state_id_and_merge_status; -- 131 MB, Never Used Indexes, table merge_requests
DROP INDEX CONCURRENTLY idx_merge_requests_on_target_project_id_and_iid_opened; -- 52 MB, index_merge_requests_on_target_project_id_and_iid, index_merge_requests_on_target_project_id_and_iid_opened, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_id_and_merge_jid; -- 4024 kB, Never Used Indexes, idx_merge_requests_on_id_and_merge_jid, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_source_project_and_branch_state_opened; -- 233 MB, idx_merge_requests_on_source_project_and_branch_state_opened, Never Used Indexes, index_merge_requests_on_source_project_id_and_source_branch, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_source_project_id_and_source_branch; -- 2704 MB, index_merge_requests_on_source_project_and_branch_state_opened, idx_merge_requests_on_source_project_and_branch_state_opened, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_state_and_merge_status; -- 403 MB, Never Used Indexes, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_target_project_id_and_iid_opened; -- 125 MB, Never Used Indexes, index_merge_requests_on_target_project_id_and_iid, idx_merge_requests_on_target_project_id_and_iid_opened, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_target_project_id_and_target_branch; -- 320 kB, index_merge_requests_on_tp_id_and_merge_commit_sha_and_id, table merge_requests
DROP INDEX CONCURRENTLY index_merge_requests_on_title; -- 3040 MB, Never Used Indexes, table merge_requests
DROP INDEX CONCURRENTLY index_namespace_aggregation_schedules_on_namespace_id; -- 2408 kB, namespace_aggregation_schedules_pkey, table namespace_aggregation_schedules
DROP INDEX CONCURRENTLY index_namespace_root_storage_statistics_on_namespace_id; -- 19 MB, namespace_root_storage_statistics_pkey, table namespace_root_storage_statistics
DROP INDEX CONCURRENTLY index_namespaces_on_created_at; -- 243 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, table namespaces
DROP INDEX CONCURRENTLY index_namespaces_on_ldap_sync_last_successful_update_at; -- 308 MB, Never Used Indexes, Never Used Indexes, table namespaces
DROP INDEX CONCURRENTLY index_namespaces_on_ldap_sync_last_update_at; -- 308 MB, Never Used Indexes, Never Used Indexes, table namespaces
DROP INDEX CONCURRENTLY index_namespaces_on_path; -- 258 MB, index_namespaces_on_runners_token, index_namespaces_on_runners_token_encrypted, table namespaces
DROP INDEX CONCURRENTLY index_namespaces_on_plan_id; -- 308 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table namespaces
DROP INDEX CONCURRENTLY index_namespaces_on_require_two_factor_authentication; -- 308 MB, Never Used Indexes, table namespaces
DROP INDEX CONCURRENTLY index_on_namespaces_lower_name; -- 343 MB, index_on_namespaces_lower_path, table namespaces
DROP INDEX CONCURRENTLY index_on_namespaces_lower_path; -- 258 MB, index_on_namespaces_lower_name, table namespaces
DROP INDEX CONCURRENTLY index_notification_settings_on_user_id; -- 448 MB, index_notifications_on_user_id_and_source_id_and_source_type, table notification_settings
DROP INDEX CONCURRENTLY index_pages_domains_need_auto_ssl_renewal; -- 328 kB, Never Used Indexes, table pages_domains
DROP INDEX CONCURRENTLY index_pages_domains_on_project_id; -- 3576 kB, index_pages_domains_on_project_id_and_enabled_until, table pages_domains
DROP INDEX CONCURRENTLY index_pages_domains_on_verified_at; -- 4584 kB, index_pages_domains_on_verified_at_and_enabled_until, table pages_domains
DROP INDEX CONCURRENTLY index_pages_domains_on_verified_at_and_enabled_until; -- 5952 kB, Never Used Indexes, table pages_domains
DROP INDEX CONCURRENTLY index_pool_repositories_on_shard_id; -- 384 kB, Never Used Indexes, table pool_repositories
DROP INDEX CONCURRENTLY index_project_custom_attributes_on_key_and_value; -- 16 kB, Never Used Indexes, table project_custom_attributes
DROP INDEX CONCURRENTLY index_project_feature_usages_on_project_id; -- 1200 kB, project_feature_usages_pkey, table project_feature_usages
DROP INDEX CONCURRENTLY index_project_mirror_data_on_last_successful_update_at; -- 462 MB, Never Used Indexes, table project_mirror_data
DROP INDEX CONCURRENTLY index_project_pages_metadata_on_project_id_and_deployed_is_true; -- 5520 kB, index_project_pages_metadata_on_project_id, table project_pages_metadata
DROP INDEX CONCURRENTLY index_project_repositories_on_shard_id; -- 278 MB, Never Used Indexes, table project_repositories
DROP INDEX CONCURRENTLY idx_repository_states_on_last_repository_verification_ran_at; -- 227 MB, Never Used Indexes, table project_repository_states
DROP INDEX CONCURRENTLY idx_repository_states_on_last_wiki_verification_ran_at; -- 243 MB, Never Used Indexes, table project_repository_states
DROP INDEX CONCURRENTLY idx_repository_states_on_repository_failure_partial; -- 16 kB, Never Used Indexes, table project_repository_states
DROP INDEX CONCURRENTLY idx_repository_states_on_wiki_failure_partial; -- 8192 bytes, Never Used Indexes, table project_repository_states
DROP INDEX CONCURRENTLY idx_repository_states_outdated_checksums; -- 23 MB, index_project_repository_states_on_project_id, idx_repository_states_on_last_repository_verification_ran_at, idx_repository_states_on_last_wiki_verification_ran_at, Never Used Indexes, table project_repository_states
DROP INDEX CONCURRENTLY idx_project_repository_check_partial; -- 1408 MB, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_on_projects_lower_path; -- 614 MB, index_projects_on_lower_name, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_on_projects_path; -- 591 MB, index_projects_on_runners_token, table projects
DROP INDEX CONCURRENTLY index_projects_on_id_partial_for_visibility; -- 74 MB, projects_pkey, table projects
DROP INDEX CONCURRENTLY index_projects_on_last_repository_check_at; -- 16 kB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_projects_on_last_repository_check_failed; -- 4758 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_projects_on_lower_name; -- 618 MB, index_on_projects_lower_path, table projects
DROP INDEX CONCURRENTLY index_projects_on_mirror_and_mirror_trigger_builds_both_true; -- 3680 kB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, projects_pkey, index_projects_on_star_count, index_projects_on_visibility_level, index_projects_on_id_partial_for_visibility, idx_projects_on_repository_storage_last_repository_updated_at, table projects
DROP INDEX CONCURRENTLY index_projects_on_mirror_last_successful_update_at; -- 4745 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_projects_on_pending_delete; -- 4888 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_projects_on_repository_storage; -- 5885 MB, idx_project_repository_check_partial, table projects
DROP INDEX CONCURRENTLY index_projects_on_runners_token; -- 4880 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY projects_requiring_code_owner_approval; -- 856 kB, Never Used Indexes, table projects
DROP INDEX CONCURRENTLY index_prometheus_alerts_on_prometheus_metric_id; -- 16 kB, Never Used Indexes, table prometheus_alerts
DROP INDEX CONCURRENTLY index_prometheus_metrics_on_group; -- 40 kB, Never Used Indexes, table prometheus_metrics
DROP INDEX CONCURRENTLY index_protected_branches_on_project_id; -- 533 MB, code_owner_approval_required, table protected_branches
DROP INDEX CONCURRENTLY index_protected_environments_on_project_id; -- 72 kB, index_protected_environments_on_project_id_and_name, table protected_environments
DROP INDEX CONCURRENTLY index_protected_tags_on_project_id; -- 576 kB, index_protected_tags_on_project_id_and_name, table protected_tags
DROP INDEX CONCURRENTLY index_remote_mirrors_on_last_successful_update_at; -- 43 MB, Never Used Indexes, table remote_mirrors
DROP INDEX CONCURRENTLY index_snippets_on_updated_at; -- 7336 kB, Never Used Indexes, table snippets
DROP INDEX CONCURRENTLY index_taggings_on_tag_id; -- 3259 MB, taggings_idx, table taggings
DROP INDEX CONCURRENTLY index_taggings_on_taggable_id_and_taggable_type; -- 5197 MB, index_taggings_on_taggable_id_and_taggable_type_and_context, table taggings
DROP INDEX CONCURRENTLY index_term_agreements_on_term_id; -- 96 MB, Never Used Indexes, table term_agreements
DROP INDEX CONCURRENTLY index_term_agreements_on_user_id; -- 81 MB, Never Used Indexes, term_agreements_unique_index, table term_agreements
DROP INDEX CONCURRENTLY index_timelogs_on_user_id; -- 34 MB, Never Used Indexes, table timelogs
DROP INDEX CONCURRENTLY index_todos_on_user_id; -- 1012 MB, index_todos_on_user_id_and_id_done, index_todos_on_user_id_and_id_pending, table todos
DROP INDEX CONCURRENTLY index_todos_on_user_id_and_id_done; -- 810 MB, index_todos_on_user_id_and_id_pending, table todos
DROP INDEX CONCURRENTLY index_todos_on_user_id_and_id_pending; -- 239 MB, index_todos_on_user_id_and_id_done, table todos
DROP INDEX CONCURRENTLY index_user_callouts_on_user_id; -- 13 MB, index_user_callouts_on_user_id_and_feature_name, Never Used Indexes, table user_callouts
DROP INDEX CONCURRENTLY index_user_statuses_on_user_id; -- 2440 kB, user_statuses_pkey, table user_statuses
DROP INDEX CONCURRENTLY index_on_users_lower_email; -- 314 MB, index_on_users_name_lower, index_on_users_lower_username, table users
DROP INDEX CONCURRENTLY index_on_users_lower_username; -- 216 MB, index_on_users_lower_email, index_on_users_name_lower, table users
DROP INDEX CONCURRENTLY index_on_users_name_lower; -- 246 MB, index_on_users_lower_username, index_on_users_lower_email, table users
DROP INDEX CONCURRENTLY index_users_on_accepted_term_id; -- 2085 MB, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table users
DROP INDEX CONCURRENTLY index_users_on_created_at; -- 310 MB, Never Used Indexes, Never Used Indexes, table users
DROP INDEX CONCURRENTLY index_users_on_email_trigram; -- 400 MB, index_users_on_username_trigram, table users
DROP INDEX CONCURRENTLY index_users_on_state; -- 2278 MB, index_users_on_state_and_internal_ee, index_users_on_state_and_internal, table users
DROP INDEX CONCURRENTLY index_users_on_state_and_internal; -- 876 MB, Never Used Indexes, index_users_on_state_and_internal_ee, Never Used Indexes, Never Used Indexes, Never Used Indexes, index_users_on_state, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, Never Used Indexes, table users
DROP INDEX CONCURRENTLY index_users_on_state_and_internal_ee; -- 874 MB, index_users_on_state_and_internal, index_users_on_state, table users
DROP INDEX CONCURRENTLY index_vulnerability_occurrences_on_vulnerability_id; -- 22 MB, Never Used Indexes, table vulnerability_occurrences
DROP INDEX CONCURRENTLY index_zoom_meetings_on_issue_id; -- 8192 bytes, index_zoom_meetings_on_issue_id_and_issue_status, table zoom_meetings

"UNDO" database migration code

CREATE INDEX CONCURRENTLY index_analytics_ca_group_stages_on_group_id ON public.analytics_cycle_analytics_group_stages USING btree (group_id); -- table analytics_cycle_analytics_group_stages
CREATE INDEX CONCURRENTLY index_analytics_ca_project_stages_on_project_id ON public.analytics_cycle_analytics_project_stages USING btree (project_id); -- table analytics_cycle_analytics_project_stages
CREATE INDEX CONCURRENTLY index_approval_merge_request_rules_1 ON public.approval_merge_request_rules USING btree (merge_request_id, code_owner); -- table approval_merge_request_rules
CREATE INDEX CONCURRENTLY index_approval_rules_code_owners_rule_type ON public.approval_merge_request_rules USING btree (merge_request_id, rule_type) WHERE (rule_type = 2); -- table approval_merge_request_rules
CREATE INDEX CONCURRENTLY index_approval_project_rules_on_project_id ON public.approval_project_rules USING btree (project_id); -- table approval_project_rules
CREATE INDEX CONCURRENTLY analytics_index_audit_events_on_created_at_and_author_id ON public.audit_events USING btree (created_at, author_id); -- table audit_events
CREATE UNIQUE INDEX index_aws_roles_on_user_id ON public.aws_roles USING btree (user_id); -- table aws_roles
CREATE INDEX CONCURRENTLY index_board_group_recent_visits_on_user_id ON public.board_group_recent_visits USING btree (user_id); -- table board_group_recent_visits
CREATE INDEX CONCURRENTLY index_board_project_recent_visits_on_user_id ON public.board_project_recent_visits USING btree (user_id); -- table board_project_recent_visits
CREATE INDEX CONCURRENTLY index_ci_builds_on_artifacts_expire_at ON public.ci_builds USING btree (artifacts_expire_at) WHERE (artifacts_file <> ''::text); -- table ci_builds
CREATE INDEX CONCURRENTLY index_ci_builds_on_project_id_for_successfull_pages_deploy ON public.ci_builds USING btree (project_id) WHERE (((type)::text = 'GenericCommitStatus'::text) AND ((stage)::text = 'deploy'::text) AND ((name)::text = 'pages:deploy'::text) AND ((status)::text = 'success'::text)); -- table ci_builds
CREATE INDEX CONCURRENTLY index_ci_builds_on_protected ON public.ci_builds USING btree (protected); -- table ci_builds
CREATE INDEX CONCURRENTLY index_ci_builds_on_queued_at ON public.ci_builds USING btree (queued_at); -- table ci_builds
CREATE INDEX CONCURRENTLY index_ci_builds_on_stage_id ON public.ci_builds USING btree (stage_id); -- table ci_builds
CREATE INDEX CONCURRENTLY index_ci_builds_metadata_on_build_id_and_has_exposed_artifacts ON public.ci_builds_metadata USING btree (build_id) WHERE (has_exposed_artifacts IS TRUE); -- table ci_builds_metadata
CREATE INDEX CONCURRENTLY index_ci_builds_metadata_on_build_id_and_interruptible ON public.ci_builds_metadata USING btree (build_id) WHERE (interruptible = true); -- table ci_builds_metadata
CREATE INDEX CONCURRENTLY index_ci_job_artifacts_on_file_store ON public.ci_job_artifacts USING btree (file_store); -- table ci_job_artifacts
CREATE INDEX CONCURRENTLY index_ci_job_artifacts_on_project_id ON public.ci_job_artifacts USING btree (project_id); -- table ci_job_artifacts
CREATE INDEX CONCURRENTLY index_ci_job_artifacts_on_project_id_for_security_reports ON public.ci_job_artifacts USING btree (project_id) WHERE (file_type = ANY (ARRAY[5, 6, 7, 8])); -- table ci_job_artifacts
CREATE INDEX CONCURRENTLY index_ci_pipeline_schedules_on_next_run_at_and_active ON public.ci_pipeline_schedules USING btree (next_run_at, active); -- table ci_pipeline_schedules
CREATE INDEX CONCURRENTLY index_ci_pipelines_on_project_id ON public.ci_pipelines USING btree (project_id); -- table ci_pipelines
CREATE INDEX CONCURRENTLY index_ci_runners_on_contacted_at ON public.ci_runners USING btree (contacted_at); -- table ci_runners
CREATE INDEX CONCURRENTLY index_ci_runners_on_is_shared ON public.ci_runners USING btree (is_shared); -- table ci_runners
CREATE INDEX CONCURRENTLY index_ci_runners_on_locked ON public.ci_runners USING btree (locked); -- table ci_runners
CREATE INDEX CONCURRENTLY index_ci_runners_on_token ON public.ci_runners USING btree (token); -- table ci_runners
CREATE INDEX CONCURRENTLY index_ci_stages_on_pipeline_id ON public.ci_stages USING btree (pipeline_id); -- table ci_stages
CREATE INDEX CONCURRENTLY index_cluster_providers_gcp_on_cloud_run ON public.cluster_providers_gcp USING btree (cloud_run); -- table cluster_providers_gcp
CREATE INDEX CONCURRENTLY index_clusters_kubernetes_namespaces_on_cluster_id ON public.clusters_kubernetes_namespaces USING btree (cluster_id); -- table clusters_kubernetes_namespaces
CREATE INDEX CONCURRENTLY index_container_repositories_on_project_id ON public.container_repositories USING btree (project_id); -- table container_repositories
CREATE INDEX CONCURRENTLY index_deploy_tokens_on_token_and_expires_at_and_id ON public.deploy_tokens USING btree (token, expires_at, id) WHERE (revoked IS FALSE); -- table deploy_tokens
CREATE INDEX CONCURRENTLY index_deployments_on_project_id_and_status ON public.deployments USING btree (project_id, status); -- table deployments
CREATE INDEX CONCURRENTLY partial_index_deployments_for_legacy_successful_deployments ON public.deployments USING btree (id) WHERE ((finished_at IS NULL) AND (status = 2)); -- table deployments
CREATE INDEX CONCURRENTLY index_design_management_designs_versions_on_design_id ON public.design_management_designs_versions USING btree (design_id); -- table design_management_designs_versions
CREATE INDEX CONCURRENTLY index_design_management_designs_versions_on_event ON public.design_management_designs_versions USING btree (event); -- table design_management_designs_versions
CREATE INDEX CONCURRENTLY index_epics_on_group_id ON public.epics USING btree (group_id); -- table epics
CREATE INDEX CONCURRENTLY index_milestone ON public.epics USING btree (milestone_id); -- table epics
CREATE INDEX CONCURRENTLY index_geo_node_namespace_links_on_geo_node_id ON public.geo_node_namespace_links USING btree (geo_node_id); -- table geo_node_namespace_links
CREATE INDEX CONCURRENTLY index_issue_links_on_source_id ON public.issue_links USING btree (source_id); -- table issue_links
CREATE INDEX CONCURRENTLY index_issue_metrics ON public.issue_metrics USING btree (issue_id); -- table issue_metrics
CREATE INDEX CONCURRENTLY index_issues_on_project_id_and_created_at_and_id_and_state ON public.issues USING btree (project_id, created_at, id, state); -- table issues
CREATE INDEX CONCURRENTLY index_issues_on_project_id_and_updated_at_and_id_and_state ON public.issues USING btree (project_id, updated_at, id, state); -- table issues
CREATE INDEX CONCURRENTLY index_issues_on_state ON public.issues USING btree (state); -- table issues
CREATE INDEX CONCURRENTLY idx_jira_connect_subscriptions_on_installation_id ON public.jira_connect_subscriptions USING btree (jira_connect_installation_id); -- table jira_connect_subscriptions
CREATE INDEX CONCURRENTLY index_label_priorities_on_priority ON public.label_priorities USING btree (priority); -- table label_priorities
CREATE INDEX CONCURRENTLY index_list_user_preferences_on_user_id ON public.list_user_preferences USING btree (user_id); -- table list_user_preferences
CREATE INDEX CONCURRENTLY index_members_on_access_level ON public.members USING btree (access_level); -- table members
CREATE INDEX CONCURRENTLY index_merge_request_assignees_on_merge_request_id ON public.merge_request_assignees USING btree (merge_request_id); -- table merge_request_assignees
CREATE INDEX CONCURRENTLY index_merge_request_diffs_on_merge_request_id_and_id ON public.merge_request_diffs USING btree (merge_request_id, id); -- table merge_request_diffs
CREATE INDEX CONCURRENTLY index_merge_request_diffs_on_merge_request_id_and_id_partial ON public.merge_request_diffs USING btree (merge_request_id, id) WHERE ((NOT stored_externally) OR (stored_externally IS NULL)); -- table merge_request_diffs
CREATE INDEX CONCURRENTLY index_merge_request_metrics ON public.merge_request_metrics USING btree (merge_request_id); -- table merge_request_metrics
CREATE INDEX CONCURRENTLY index_merge_request_metrics_on_first_deployed_to_production_at ON public.merge_request_metrics USING btree (first_deployed_to_production_at); -- table merge_request_metrics
CREATE INDEX CONCURRENTLY index_merge_request_metrics_on_latest_closed_at ON public.merge_request_metrics USING btree (latest_closed_at) WHERE (latest_closed_at IS NOT NULL); -- table merge_request_metrics
CREATE INDEX CONCURRENTLY idx_merge_requests_on_id_and_merge_jid ON public.merge_requests USING btree (id, merge_jid) WHERE ((merge_jid IS NOT NULL) AND (state_id = 4)); -- table merge_requests
CREATE INDEX CONCURRENTLY idx_merge_requests_on_source_project_and_branch_state_opened ON public.merge_requests USING btree (source_project_id, source_branch) WHERE (state_id = 1); -- table merge_requests
CREATE INDEX CONCURRENTLY idx_merge_requests_on_state_id_and_merge_status ON public.merge_requests USING btree (state_id, merge_status) WHERE ((state_id = 1) AND ((merge_status)::text = 'can_be_merged'::text)); -- table merge_requests
CREATE INDEX CONCURRENTLY idx_merge_requests_on_target_project_id_and_iid_opened ON public.merge_requests USING btree (target_project_id, iid) WHERE (state_id = 1); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_id_and_merge_jid ON public.merge_requests USING btree (id, merge_jid) WHERE ((merge_jid IS NOT NULL) AND ((state)::text = 'locked'::text)); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_source_project_and_branch_state_opened ON public.merge_requests USING btree (source_project_id, source_branch) WHERE ((state)::text = 'opened'::text); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_source_project_id_and_source_branch ON public.merge_requests USING btree (source_project_id, source_branch); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_state_and_merge_status ON public.merge_requests USING btree (state, merge_status) WHERE (((state)::text = 'opened'::text) AND ((merge_status)::text = 'can_be_merged'::text)); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_target_project_id_and_iid_opened ON public.merge_requests USING btree (target_project_id, iid) WHERE ((state)::text = 'opened'::text); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_target_project_id_and_target_branch ON public.merge_requests USING btree (target_project_id, target_branch) WHERE ((state_id = 1) AND (merge_when_pipeline_succeeds = true)); -- table merge_requests
CREATE INDEX CONCURRENTLY index_merge_requests_on_title ON public.merge_requests USING btree (title); -- table merge_requests
CREATE UNIQUE INDEX index_namespace_aggregation_schedules_on_namespace_id ON public.namespace_aggregation_schedules USING btree (namespace_id); -- table namespace_aggregation_schedules
CREATE UNIQUE INDEX index_namespace_root_storage_statistics_on_namespace_id ON public.namespace_root_storage_statistics USING btree (namespace_id); -- table namespace_root_storage_statistics
CREATE INDEX CONCURRENTLY index_namespaces_on_created_at ON public.namespaces USING btree (created_at); -- table namespaces
CREATE INDEX CONCURRENTLY index_namespaces_on_ldap_sync_last_successful_update_at ON public.namespaces USING btree (ldap_sync_last_successful_update_at); -- table namespaces
CREATE INDEX CONCURRENTLY index_namespaces_on_ldap_sync_last_update_at ON public.namespaces USING btree (ldap_sync_last_update_at); -- table namespaces
CREATE INDEX CONCURRENTLY index_namespaces_on_path ON public.namespaces USING btree (path); -- table namespaces
CREATE INDEX CONCURRENTLY index_namespaces_on_plan_id ON public.namespaces USING btree (plan_id); -- table namespaces
CREATE INDEX CONCURRENTLY index_namespaces_on_require_two_factor_authentication ON public.namespaces USING btree (require_two_factor_authentication); -- table namespaces
CREATE INDEX CONCURRENTLY index_on_namespaces_lower_name ON public.namespaces USING btree (lower((name)::text)); -- table namespaces
CREATE INDEX CONCURRENTLY index_on_namespaces_lower_path ON public.namespaces USING btree (lower((path)::text)); -- table namespaces
CREATE INDEX CONCURRENTLY index_notification_settings_on_user_id ON public.notification_settings USING btree (user_id); -- table notification_settings
CREATE INDEX CONCURRENTLY index_pages_domains_need_auto_ssl_renewal ON public.pages_domains USING btree (certificate_source, certificate_valid_not_after) WHERE (auto_ssl_enabled = true); -- table pages_domains
CREATE INDEX CONCURRENTLY index_pages_domains_on_project_id ON public.pages_domains USING btree (project_id); -- table pages_domains
CREATE INDEX CONCURRENTLY index_pages_domains_on_verified_at ON public.pages_domains USING btree (verified_at); -- table pages_domains
CREATE INDEX CONCURRENTLY index_pages_domains_on_verified_at_and_enabled_until ON public.pages_domains USING btree (verified_at, enabled_until); -- table pages_domains
CREATE INDEX CONCURRENTLY index_pool_repositories_on_shard_id ON public.pool_repositories USING btree (shard_id); -- table pool_repositories
CREATE INDEX CONCURRENTLY index_project_custom_attributes_on_key_and_value ON public.project_custom_attributes USING btree (key, value); -- table project_custom_attributes
CREATE INDEX CONCURRENTLY index_project_feature_usages_on_project_id ON public.project_feature_usages USING btree (project_id); -- table project_feature_usages
CREATE INDEX CONCURRENTLY index_project_mirror_data_on_last_successful_update_at ON public.project_mirror_data USING btree (last_successful_update_at); -- table project_mirror_data
CREATE INDEX CONCURRENTLY index_project_pages_metadata_on_project_id_and_deployed_is_true ON public.project_pages_metadata USING btree (project_id) WHERE (deployed = true); -- table project_pages_metadata
CREATE INDEX CONCURRENTLY index_project_repositories_on_shard_id ON public.project_repositories USING btree (shard_id); -- table project_repositories
CREATE INDEX CONCURRENTLY idx_repository_states_on_last_repository_verification_ran_at ON public.project_repository_states USING btree (project_id, last_repository_verification_ran_at) WHERE ((repository_verification_checksum IS NOT NULL) AND (last_repository_verification_failure IS NULL)); -- table project_repository_states
CREATE INDEX CONCURRENTLY idx_repository_states_on_last_wiki_verification_ran_at ON public.project_repository_states USING btree (project_id, last_wiki_verification_ran_at) WHERE ((wiki_verification_checksum IS NOT NULL) AND (last_wiki_verification_failure IS NULL)); -- table project_repository_states
CREATE INDEX CONCURRENTLY idx_repository_states_on_repository_failure_partial ON public.project_repository_states USING btree (last_repository_verification_failure) WHERE (last_repository_verification_failure IS NOT NULL); -- table project_repository_states
CREATE INDEX CONCURRENTLY idx_repository_states_on_wiki_failure_partial ON public.project_repository_states USING btree (last_wiki_verification_failure) WHERE (last_wiki_verification_failure IS NOT NULL); -- table project_repository_states
CREATE INDEX CONCURRENTLY idx_repository_states_outdated_checksums ON public.project_repository_states USING btree (project_id) WHERE (((repository_verification_checksum IS NULL) AND (last_repository_verification_failure IS NULL)) OR ((wiki_verification_checksum IS NULL) AND (last_wiki_verification_failure IS NULL))); -- table project_repository_states
CREATE INDEX CONCURRENTLY idx_project_repository_check_partial ON public.projects USING btree (repository_storage, created_at) WHERE (last_repository_check_at IS NULL); -- table projects
CREATE INDEX CONCURRENTLY index_on_projects_lower_path ON public.projects USING btree (lower((path)::text)); -- table projects
CREATE INDEX CONCURRENTLY index_on_projects_path ON public.projects USING btree (path); -- table projects
CREATE UNIQUE INDEX index_projects_on_id_partial_for_visibility ON public.projects USING btree (id) WHERE (visibility_level = ANY (ARRAY[10, 20])); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_last_repository_check_at ON public.projects USING btree (last_repository_check_at) WHERE (last_repository_check_at IS NOT NULL); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_last_repository_check_failed ON public.projects USING btree (last_repository_check_failed); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_lower_name ON public.projects USING btree (lower((name)::text)); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_mirror_and_mirror_trigger_builds_both_true ON public.projects USING btree (id) WHERE ((mirror IS TRUE) AND (mirror_trigger_builds IS TRUE)); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_mirror_last_successful_update_at ON public.projects USING btree (mirror_last_successful_update_at); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_pending_delete ON public.projects USING btree (pending_delete); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_repository_storage ON public.projects USING btree (repository_storage); -- table projects
CREATE INDEX CONCURRENTLY index_projects_on_runners_token ON public.projects USING btree (runners_token); -- table projects
CREATE INDEX CONCURRENTLY projects_requiring_code_owner_approval ON public.projects USING btree (archived, pending_delete, merge_requests_require_code_owner_approval) WHERE ((pending_delete = false) AND (archived = false) AND (merge_requests_require_code_owner_approval = true)); -- table projects
CREATE INDEX CONCURRENTLY index_prometheus_alerts_on_prometheus_metric_id ON public.prometheus_alerts USING btree (prometheus_metric_id); -- table prometheus_alerts
CREATE INDEX CONCURRENTLY index_prometheus_metrics_on_group ON public.prometheus_metrics USING btree ("group"); -- table prometheus_metrics
CREATE INDEX CONCURRENTLY index_protected_branches_on_project_id ON public.protected_branches USING btree (project_id); -- table protected_branches
CREATE INDEX CONCURRENTLY index_protected_environments_on_project_id ON public.protected_environments USING btree (project_id); -- table protected_environments
CREATE INDEX CONCURRENTLY index_protected_tags_on_project_id ON public.protected_tags USING btree (project_id); -- table protected_tags
CREATE INDEX CONCURRENTLY index_remote_mirrors_on_last_successful_update_at ON public.remote_mirrors USING btree (last_successful_update_at); -- table remote_mirrors
CREATE INDEX CONCURRENTLY index_snippets_on_updated_at ON public.snippets USING btree (updated_at); -- table snippets
CREATE INDEX CONCURRENTLY index_taggings_on_tag_id ON public.taggings USING btree (tag_id); -- table taggings
CREATE INDEX CONCURRENTLY index_taggings_on_taggable_id_and_taggable_type ON public.taggings USING btree (taggable_id, taggable_type); -- table taggings
CREATE INDEX CONCURRENTLY index_term_agreements_on_term_id ON public.term_agreements USING btree (term_id); -- table term_agreements
CREATE INDEX CONCURRENTLY index_term_agreements_on_user_id ON public.term_agreements USING btree (user_id); -- table term_agreements
CREATE INDEX CONCURRENTLY index_timelogs_on_user_id ON public.timelogs USING btree (user_id); -- table timelogs
CREATE INDEX CONCURRENTLY index_todos_on_user_id ON public.todos USING btree (user_id); -- table todos
CREATE INDEX CONCURRENTLY index_todos_on_user_id_and_id_done ON public.todos USING btree (user_id, id) WHERE ((state)::text = 'done'::text); -- table todos
CREATE INDEX CONCURRENTLY index_todos_on_user_id_and_id_pending ON public.todos USING btree (user_id, id) WHERE ((state)::text = 'pending'::text); -- table todos
CREATE INDEX CONCURRENTLY index_user_callouts_on_user_id ON public.user_callouts USING btree (user_id); -- table user_callouts
CREATE INDEX CONCURRENTLY index_user_statuses_on_user_id ON public.user_statuses USING btree (user_id); -- table user_statuses
CREATE INDEX CONCURRENTLY index_on_users_lower_email ON public.users USING btree (lower((email)::text)); -- table users
CREATE INDEX CONCURRENTLY index_on_users_lower_username ON public.users USING btree (lower((username)::text)); -- table users
CREATE INDEX CONCURRENTLY index_on_users_name_lower ON public.users USING btree (lower((name)::text)); -- table users
CREATE INDEX CONCURRENTLY index_users_on_accepted_term_id ON public.users USING btree (accepted_term_id); -- table users
CREATE INDEX CONCURRENTLY index_users_on_created_at ON public.users USING btree (created_at); -- table users
CREATE INDEX CONCURRENTLY index_users_on_email_trigram ON public.users USING gin (email gin_trgm_ops); -- table users
CREATE INDEX CONCURRENTLY index_users_on_state ON public.users USING btree (state); -- table users
CREATE INDEX CONCURRENTLY index_users_on_state_and_internal ON public.users USING btree (state) WHERE (ghost IS NOT TRUE); -- table users
CREATE INDEX CONCURRENTLY index_users_on_state_and_internal_ee ON public.users USING btree (state) WHERE ((ghost IS NOT TRUE) AND (bot_type IS NULL)); -- table users
CREATE INDEX CONCURRENTLY index_vulnerability_occurrences_on_vulnerability_id ON public.vulnerability_occurrences USING btree (vulnerability_id); -- table vulnerability_occurrences
CREATE INDEX CONCURRENTLY index_zoom_meetings_on_issue_id ON public.zoom_meetings USING btree (issue_id); -- table zoom_meetings

Table of contents

H003 Non-indexed Foreign Keys

Observations

Data collected: 2019-11-11 13:41:27 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Num	Schema name	Table name	FK name	Issue	Table mb	writes	Table scans	Parent name	Parent mb	Parent writes	Cols list	Indexdef
1	`public`	`design_management_versions`	`fk_c1440b4896`	no index	0	440	2001	users	2107	10384118	[author_id]
2	`public`	`ci_builds`	`fk_87f4cefcda`	questionable index	335572	44327996	0	ci_pipelines	18773	12025071	[upstream_pipeline_id]	CREATE INDEX index_ci_builds_on_upstream_pipeline_id ON public.ci_builds USING btree (upstream_pipeline_id) WHERE (upstream_pipeline_id IS NOT NULL)
3	`public`	`ci_builds`	`fk_d3130c9a7f`	questionable index	335572	44327996	0	ci_pipelines	18773	12025071	[commit_id]	CREATE INDEX index_ci_builds_on_commit_id_and_artifacts_expireatandidpartial ON public.ci_builds USING btree (commit_id, artifacts_expire_at, id) WHERE (((type)::text = 'Ci::Build'::text) AND ((retried = false) OR (retried IS NULL)) AND ((name)::text = ANY ((ARRAY['sast'::character varying, 'dependency_scanning'::character varying, 'sast:container'::character varying, 'container_scanning'::character varying, 'dast'::character varying])::text[])))
4	`public`	`ci_builds`	`fk_d3130c9a7f`	questionable index	335572	44327996	0	ci_pipelines	18773	12025071	[commit_id]	CREATE INDEX index_ci_builds_on_commit_id_and_type_and_ref ON public.ci_builds USING btree (commit_id, type, ref)
5	`public`	`ci_builds`	`fk_d3130c9a7f`	questionable index	335572	44327996	0	ci_pipelines	18773	12025071	[commit_id]	CREATE INDEX index_ci_builds_on_commit_id_and_type_and_name_and_ref ON public.ci_builds USING btree (commit_id, type, name, ref)
6	`public`	`ci_builds`	`fk_d3130c9a7f`	questionable index	335572	44327996	0	ci_pipelines	18773	12025071	[commit_id]	CREATE INDEX index_ci_builds_on_commit_id_and_status_and_type ON public.ci_builds USING btree (commit_id, status, type)
7	`public`	`ci_builds`	`fk_d3130c9a7f`	questionable index	335572	44327996	0	ci_pipelines	18773	12025071	[commit_id]	CREATE INDEX index_ci_builds_on_commit_id_and_stage_idx_and_created_at ON public.ci_builds USING btree (commit_id, stage_idx, created_at)
8	`public`	`events`	`fk_61fbf6ca48`	questionable index	32054	13151264	0	namespaces	750	204707	[group_id]	CREATE INDEX index_events_on_group_id_partial ON public.events USING btree (group_id) WHERE (group_id IS NOT NULL)
9	`public`	`ci_pipelines`	`fk_190998ef09`	questionable index	18773	12025071	0	external_pull_requests	2	23394	[external_pull_request_id]	CREATE INDEX index_ci_pipelines_on_external_pull_request_id ON public.ci_pipelines USING btree (external_pull_request_id) WHERE (external_pull_request_id IS NOT NULL)
10	`public`	`ci_pipelines`	`fk_a23be95014`	questionable index	18773	12025071	0	merge_requests	15123	18177781	[merge_request_id]	CREATE INDEX index_ci_pipelines_on_merge_request_id ON public.ci_pipelines USING btree (merge_request_id) WHERE (merge_request_id IS NOT NULL)
11	`public`	`merge_requests`	`fk_641731faff`	questionable index	15123	18177781	6	users	2107	10384118	[updated_by_id]	CREATE INDEX index_merge_requests_on_updated_by_id ON public.merge_requests USING btree (updated_by_id) WHERE (updated_by_id IS NOT NULL)
12	`public`	`merge_requests`	`fk_ad525e1f87`	questionable index	15123	18177781	6	users	2107	10384118	[merge_user_id]	CREATE INDEX index_merge_requests_on_merge_user_id ON public.merge_requests USING btree (merge_user_id) WHERE (merge_user_id IS NOT NULL)
13	`public`	`issues`	`fk_9c4516d665`	questionable index	13717	3802571	2078	issues	13717	3802571	[duplicated_to_id]	CREATE INDEX index_issues_on_duplicated_to_id ON public.issues USING btree (duplicated_to_id) WHERE (duplicated_to_id IS NOT NULL)
14	`public`	`issues`	`fk_a194299be1`	questionable index	13717	3802571	2078	issues	13717	3802571	[moved_to_id]	CREATE INDEX index_issues_on_moved_to_id ON public.issues USING btree (moved_to_id) WHERE (moved_to_id IS NOT NULL)
15	`public`	`issues`	`fk_ffed080f01`	questionable index	13717	3802571	2078	users	2107	10384118	[updated_by_id]	CREATE INDEX index_issues_on_updated_by_id ON public.issues USING btree (updated_by_id) WHERE (updated_by_id IS NOT NULL)
16	`public`	`system_note_metadata`	`fk_fbd87415c9`	questionable index	8413	4261347	0	description_versions	0	0	[description_version_id]	CREATE UNIQUE INDEX index_system_note_metadata_on_description_version_id ON public.system_note_metadata USING btree (description_version_id) WHERE (description_version_id IS NOT NULL)
17	`public`	`projects`	`fk_0a31cca0b8`	questionable index	3018	13641217	67365	users	2107	10384118	[marked_for_deletion_by_user_id]	CREATE INDEX index_projects_on_marked_for_deletion_by_user_id ON public.projects USING btree (marked_for_deletion_by_user_id) WHERE (marked_for_deletion_by_user_id IS NOT NULL)
18	`public`	`projects`	`fk_6e5c14658a`	questionable index	3018	13641217	67365	pool_repositories	2	6137	[pool_repository_id]	CREATE INDEX index_projects_on_pool_repository_id ON public.projects USING btree (pool_repository_id) WHERE (pool_repository_id IS NOT NULL)
19	`public`	`project_authorizations`	`fk_rails_11e7aa3ed9`	questionable index	1924	5209307	0	users	2107	10384118	[user_id]	CREATE UNIQUE INDEX index_project_authorizations_on_user_id_project_id_access_level ON public.project_authorizations USING btree (user_id, project_id, access_level)
20	`public`	`vulnerability_occurrences`	`fk_rails_90fed4faba`	questionable index	1065	80962	0	projects	3018	13641217	[project_id]	CREATE UNIQUE INDEX index_vulnerability_occurrences_on_unique_keys ON public.vulnerability_occurrences USING btree (project_id, primary_identifier_id, location_fingerprint, scanner_id)

Conclusions

Recommendations

Table of contents

K001 Globally Aggregated Query Metrics

Observations

Data collected: 2019-11-11 13:48:34 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Start: 2019-11-11T13:05:37.597123+00:00
End: 2019-11-11T13:41:28.060124+00:00
Period seconds: 2150.463
Period age: 00:35:50.463001

Error (calls): 0.00 (0.00%)
Error (total time): 0.00 (0.00%)

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
33,115,713 15.40K/sec 1.00/call 100.00%	33,784,887.09 ms 15710.518 ms/sec 1.020 ms/call 100.00%	179,070,413 83.28K/sec 5.41/call 100.00%	1,408,113,742 blks 654.80K blks/sec 42.52 blks/call 100.00%	832,994 blks 387.36 blks/sec 0.03 blks/call 100.00%	1,340,246 blks 623.24 blks/sec 0.04 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	354,042.07 ms 164.635 ms/sec 0.011 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica servers:

Replica (`10.220.16.101`)

Nothing found

Replica (`10.220.16.102`)

Start: 2019-11-11T13:05:32.574522+00:00
End: 2019-11-11T13:36:01.607707+00:00
Period seconds: 1829.03319
Period age: 00:30:29.033185

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
9,779,648 5.35K/sec 1.00/call 100.00%	2,822,533.56 ms 1543.183 ms/sec 0.289 ms/call 100.00%	35,258,834 19.28K/sec 3.61/call 100.00%	1,236,960,822 blks 676.30K blks/sec 126.48 blks/call 100.00%	1,467,145 blks 802.14 blks/sec 0.15 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	71,321.82 ms 38.994 ms/sec 0.007 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.103`)

Start: 2019-11-11T13:05:33.920039+00:00
End: 2019-11-11T13:36:46.045187+00:00
Period seconds: 1872.12515
Period age: 00:31:12.125148

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
9,670,366 5.17K/sec 1.00/call 100.00%	2,710,430.63 ms 1447.783 ms/sec 0.280 ms/call 100.00%	34,505,032 18.44K/sec 3.57/call 100.00%	1,198,322,792 blks 640.09K blks/sec 123.92 blks/call 100.00%	1,606,382 blks 858.05 blks/sec 0.17 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	141,794.17 ms 75.740 ms/sec 0.015 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.104`)

Start: 2019-11-11T13:05:35.26518+00:00
End: 2019-11-11T13:37:44.522444+00:00
Period seconds: 1929.25726
Period age: 00:32:09.257264

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
10,049,280 5.21K/sec 1.00/call 100.00%	2,560,526.29 ms 1327.208 ms/sec 0.255 ms/call 100.00%	37,454,923 19.42K/sec 3.73/call 100.00%	1,226,674,657 blks 635.83K blks/sec 122.07 blks/call 100.00%	1,722,661 blks 892.91 blks/sec 0.17 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	127,414.07 ms 66.043 ms/sec 0.013 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.105`)

Start: 2019-11-11T13:05:36.541154+00:00
End: 2019-11-11T13:38:58.630579+00:00
Period seconds: 2002.08943
Period age: 00:33:22.089425

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
9,810,924 4.91K/sec 1.00/call 100.00%	2,697,335.77 ms 1347.260 ms/sec 0.275 ms/call 100.00%	36,848,710 18.41K/sec 3.76/call 100.00%	1,358,490,386 blks 678.54K blks/sec 138.47 blks/call 100.00%	1,698,920 blks 848.57 blks/sec 0.17 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	134,930.97 ms 67.395 ms/sec 0.014 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.107`)

Start: 2019-11-11T13:05:38.708195+00:00
End: 2019-11-11T13:43:26.762667+00:00
Period seconds: 2268.05448
Period age: 00:37:48.054472

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
12,077,710 5.33K/sec 1.00/call 100.00%	3,650,037.30 ms 1609.325 ms/sec 0.302 ms/call 100.00%	42,985,914 18.96K/sec 3.56/call 100.00%	1,754,676,140 blks 773.65K blks/sec 145.28 blks/call 100.00%	1,994,914 blks 879.57 blks/sec 0.17 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	99,321.90 ms 43.792 ms/sec 0.008 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.108`)

Start: 2019-11-11T13:05:39.790748+00:00
End: 2019-11-11T13:45:48.057645+00:00
Period seconds: 2408.2669
Period age: 00:40:08.266897

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
13,007,439 5.41K/sec 1.00/call 100.00%	3,741,152.33 ms 1553.463 ms/sec 0.288 ms/call 100.00%	46,607,334 19.36K/sec 3.58/call 100.00%	1,694,365,960 blks 703.57K blks/sec 130.26 blks/call 100.00%	1,970,735 blks 818.32 blks/sec 0.15 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	145,074.06 ms 60.240 ms/sec 0.011 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.109`)

Start: 2019-11-11T13:05:40.877736+00:00
End: 2019-11-11T13:48:35.192983+00:00
Period seconds: 2574.31524
Period age: 00:42:54.315247

Calls	Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
12,766,399 4.96K/sec 1.00/call 100.00%	4,202,516.49 ms 1632.479 ms/sec 0.329 ms/call 100.00%	49,077,587 19.07K/sec 3.84/call 100.00%	1,802,375,926 blks 700.14K blks/sec 141.18 blks/call 100.00%	2,338,142 blks 908.26 blks/sec 0.18 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	144,273.15 ms 56.043 ms/sec 0.011 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Conclusions

Recommendations

Table of contents

K002 Workload Type ("The First Word" Analysis)

Observations

Data collected: 2019-11-11 13:48:34 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Start: 2019-11-11T13:05:37.597123+00:00
End: 2019-11-11T13:41:28.060124+00:00
Period seconds: 2150.463
Period age: 00:35:50.463001

Error (calls): 0.00 (0.00%)
Error (total time): 0.00 (0.00%)

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	update	678,859 315.68/sec 1.00/call 2.05%	29,429,241.37 ms 13685.072 ms/sec 43.351 ms/call 87.11%	543,593 252.78/sec 0.80/call 0.30%	25,742,495 blks 11.98K blks/sec 37.92 blks/call 1.83%	73,170 blks 34.03 blks/sec 0.11 blks/call 8.78%	854,544 blks 397.38 blks/sec 1.26 blks/call 63.76%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	14,917.41 ms 6.937 ms/sec 0.022 ms/call 4.21%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	select	27,643,235 12.86K/sec 1.00/call 83.47%	4,022,096.99 ms 1870.340 ms/sec 0.146 ms/call 11.91%	157,710,198 73.34K/sec 5.71/call 88.07%	1,314,813,733 blks 611.41K blks/sec 47.56 blks/call 93.37%	652,648 blks 303.49 blks/sec 0.02 blks/call 78.35%	98,672 blks 45.88 blks/sec 0.00 blks/call 7.36%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	268,656.46 ms 124.930 ms/sec 0.010 ms/call 75.88%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	with	205,628 95.62/sec 1.00/call 0.62%	147,901.25 ms 68.776 ms/sec 0.719 ms/call 0.44%	20,507,277 9.54K/sec 99.73/call 11.45%	57,000,072 blks 26.51K blks/sec 277.20 blks/call 4.05%	2,339 blks 1.09 blks/sec 0.01 blks/call 0.28%	473 blks 0.22 blks/sec 0.00 blks/call 0.04%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	987.91 ms 0.459 ms/sec 0.005 ms/call 0.28%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	insert	144,729 67.30/sec 1.00/call 0.44%	137,975.21 ms 64.161 ms/sec 0.953 ms/call 0.41%	286,585 133.27/sec 1.98/call 0.16%	10,441,986 blks 4.86K blks/sec 72.15 blks/call 0.74%	103,727 blks 48.23 blks/sec 0.72 blks/call 12.45%	383,382 blks 178.28 blks/sec 2.65 blks/call 28.61%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	69,451.09 ms 32.296 ms/sec 0.480 ms/call 19.62%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	set	4,420,502 2.06K/sec 1.00/call 13.35%	44,626.28 ms 20.752 ms/sec 0.010 ms/call 0.13%	0 0.00/sec 0.00/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	select ... for [no key] update	22,760 10.58/sec 1.00/call 0.07%	3,045.99 ms 1.416 ms/sec 0.134 ms/call 0.01%	22,760 10.58/sec 1.00/call 0.01%	115,456 blks 53.69 blks/sec 5.07 blks/call 0.01%	1,110 blks 0.52 blks/sec 0.05 blks/call 0.13%	3,175 blks 1.48 blks/sec 0.14 blks/call 0.24%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	29.20 ms 0.014 ms/sec 0.001 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica servers:

Replica (`10.220.16.101`)

Nothing found

Replica (`10.220.16.102`)

Start: 2019-11-11T13:05:32.574522+00:00
End: 2019-11-11T13:36:01.607707+00:00
Period seconds: 1829.03319
Period age: 00:30:29.033185

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	9,296,285 5.09K/sec 1.00/call 95.06%	2,562,191.16 ms 1400.845 ms/sec 0.276 ms/call 90.78%	34,415,612 18.82K/sec 3.70/call 97.61%	1,134,771,821 blks 620.43K blks/sec 122.07 blks/call 91.74%	1,467,121 blks 802.13 blks/sec 0.16 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	71,320.53 ms 38.994 ms/sec 0.008 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	483,363 264.27/sec 1.00/call 4.94%	260,342.40 ms 142.339 ms/sec 0.539 ms/call 9.22%	843,222 461.02/sec 1.74/call 2.39%	102,189,001 blks 55.88K blks/sec 211.41 blks/call 8.26%	24 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1.30 ms 0.001 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.103`)

Start: 2019-11-11T13:05:33.920039+00:00
End: 2019-11-11T13:36:46.045187+00:00
Period seconds: 1872.12515
Period age: 00:31:12.125148

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	8,681,070 4.64K/sec 1.00/call 89.77%	2,447,546.55 ms 1307.363 ms/sec 0.282 ms/call 90.30%	33,642,800 17.98K/sec 3.88/call 97.50%	1,102,860,822 blks 589.10K blks/sec 127.04 blks/call 92.03%	1,606,358 blks 858.04 blks/sec 0.19 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	141,792.28 ms 75.739 ms/sec 0.016 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	494,677 264.23/sec 1.00/call 5.12%	260,392.84 ms 139.089 ms/sec 0.526 ms/call 9.61%	862,232 460.56/sec 1.74/call 2.50%	95,461,970 blks 51.00K blks/sec 192.98 blks/call 7.97%	24 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1.89 ms 0.001 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	set	494,619 264.20/sec 1.00/call 5.11%	2,491.24 ms 1.331 ms/sec 0.005 ms/call 0.09%	0 0.00/sec 0.00/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.104`)

Start: 2019-11-11T13:05:35.26518+00:00
End: 2019-11-11T13:37:44.522444+00:00
Period seconds: 1929.25726
Period age: 00:32:09.257264

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	9,542,147 4.95K/sec 1.00/call 94.95%	2,359,205.59 ms 1222.857 ms/sec 0.247 ms/call 92.14%	36,568,093 18.96K/sec 3.83/call 97.63%	1,149,863,857 blks 596.02K blks/sec 120.50 blks/call 93.74%	1,722,646 blks 892.91 blks/sec 0.18 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	127,411.51 ms 66.042 ms/sec 0.013 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	507,133 262.86/sec 1.00/call 5.05%	201,320.70 ms 104.351 ms/sec 0.397 ms/call 7.86%	886,830 459.67/sec 1.75/call 2.37%	76,810,800 blks 39.82K blks/sec 151.46 blks/call 6.26%	15 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2.57 ms 0.001 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.105`)

Start: 2019-11-11T13:05:36.541154+00:00
End: 2019-11-11T13:38:58.630579+00:00
Period seconds: 2002.08943
Period age: 00:33:22.089425

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	9,287,134 4.64K/sec 1.00/call 94.66%	2,493,608.83 ms 1245.503 ms/sec 0.269 ms/call 92.45%	35,932,602 17.95K/sec 3.87/call 97.51%	1,277,655,585 blks 638.17K blks/sec 137.57 blks/call 94.05%	1,698,898 blks 848.56 blks/sec 0.18 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	134,927.88 ms 67.394 ms/sec 0.015 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	523,790 261.62/sec 1.00/call 5.34%	203,726.93 ms 101.757 ms/sec 0.389 ms/call 7.55%	916,108 457.58/sec 1.75/call 2.49%	80,834,801 blks 40.38K blks/sec 154.33 blks/call 5.95%	22 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	3.08 ms 0.002 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.107`)

Start: 2019-11-11T13:05:38.708195+00:00
End: 2019-11-11T13:43:26.762667+00:00
Period seconds: 2268.05448
Period age: 00:37:48.054472

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	11,481,534 5.07K/sec 1.00/call 95.06%	3,411,305.19 ms 1504.067 ms/sec 0.297 ms/call 93.46%	41,947,832 18.50K/sec 3.65/call 97.59%	1,648,868,866 blks 727.00K blks/sec 143.61 blks/call 93.97%	1,994,901 blks 879.56 blks/sec 0.17 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	99,320.99 ms 43.791 ms/sec 0.009 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	596,176 262.86/sec 1.00/call 4.94%	238,732.11 ms 105.259 ms/sec 0.400 ms/call 6.54%	1,038,082 457.70/sec 1.74/call 2.41%	105,807,274 blks 46.66K blks/sec 177.48 blks/call 6.03%	13 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.91 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.108`)

Start: 2019-11-11T13:05:39.790748+00:00
End: 2019-11-11T13:45:48.057645+00:00
Period seconds: 2408.2669
Period age: 00:40:08.266897

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	12,366,583 5.14K/sec 1.00/call 95.07%	3,413,281.32 ms 1417.319 ms/sec 0.276 ms/call 91.24%	45,493,472 18.90K/sec 3.68/call 97.61%	1,556,738,224 blks 646.42K blks/sec 125.88 blks/call 91.88%	1,970,711 blks 818.31 blks/sec 0.16 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	145,073.68 ms 60.240 ms/sec 0.012 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	640,856 266.11/sec 1.00/call 4.93%	327,871.01 ms 136.144 ms/sec 0.512 ms/call 8.76%	1,113,862 462.52/sec 1.74/call 2.39%	137,627,736 blks 57.15K blks/sec 214.76 blks/call 8.12%	24 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.39 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.109`)

Start: 2019-11-11T13:05:40.877736+00:00
End: 2019-11-11T13:48:35.192983+00:00
Period seconds: 2574.31524
Period age: 00:42:54.315247

#	Workload type	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	select	12,084,183 4.70K/sec 1.00/call 94.66%	3,823,645.25 ms 1485.306 ms/sec 0.316 ms/call 90.98%	47,888,848 18.61K/sec 3.96/call 97.58%	1,669,078,970 blks 648.36K blks/sec 138.12 blks/call 92.60%	2,338,116 blks 908.25 blks/sec 0.19 blks/call 100.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	144,272.67 ms 56.043 ms/sec 0.012 ms/call 100.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	with	682,216 265.01/sec 1.00/call 5.34%	378,871.23 ms 147.174 ms/sec 0.555 ms/call 9.02%	1,188,739 461.77/sec 1.74/call 2.42%	133,296,956 blks 51.78K blks/sec 195.39 blks/call 7.40%	26 blks 0.01 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.48 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Conclusions

Recommendations

Table of contents

K003 Top-20 Queries by total_time

Observations

Data collected: 2019-11-11 13:48:34 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Start: 2019-11-11T13:05:37.597123+00:00
End: 2019-11-11T13:41:28.060124+00:00
Period seconds: 2150.463
Period age: 00:35:50.463001

Error (calls): 0.00 (0.00%)
Error (total time): 0.00 (0.00%)

The list is limited to 20 items.

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	UPDATE "ci_builds" SET "runner_id" = ?, "status" = ?, "started_at" = ?, "updated_at" = ?, "lock_version" = ? WHERE "ci_builds"."type" IN (?) AND "ci_builds"."lock_version" = ? AND "ci_builds"."id" = ? Full query	88,135 40.98/sec 1.00/call 0.27%	25,871,225.82 ms 12030.538 ms/sec 293.541 ms/call 76.58%	25,630 11.92/sec 0.29/call 0.01%	2,858,856 blks 1.33K blks/sec 32.44 blks/call 0.20%	7,047 blks 3.28 blks/sec 0.08 blks/call 0.85%	29,176 blks 13.57 blks/sec 0.33 blks/call 2.18%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	3,001.20 ms 1.396 ms/sec 0.034 ms/call 0.85%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	UPDATE "project_statistics" SET build_artifacts_size = COALESCE( build_artifacts_size, ?) + (?), storage_size = COALESCE(storage_size, ?) + (?) WHERE "project_statistics"."project_id" = ? Full query	39,230 18.24/sec 1.00/call 0.12%	2,155,101.98 ms 1002.157 ms/sec 54.935 ms/call 6.38%	39,230 18.24/sec 1.00/call 0.02%	1,389,078 blks 645.94 blks/sec 35.41 blks/call 0.10%	603 blks 0.28 blks/sec 0.02 blks/call 0.07%	32,310 blks 15.02 blks/sec 0.82 blks/call 2.41%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	51.55 ms 0.024 ms/sec 0.001 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	177 0.08/sec 1.00/call 0.00%	694,233.51 ms 322.830 ms/sec 3922.223 ms/call 2.05%	177 0.08/sec 1.00/call 0.00%	68,379,178 blks 31.80K blks/sec 386.33K blks/call 4.86%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	5,399 blks 2.51 blks/sec 30.50 blks/call 0.40%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	SELECT "projects".* FROM "projects" INNER JOIN project_mirror_data import_state ON import_state.project_id = projects.id WHERE "projects"."mirror" = ? AND "import_state"."status" NOT IN (?, ?) AND (import_state.next_execution_timestamp <= ?) AND (import_state.retry_count <= ?) ORDER BY import_state.next_execution_timestamp LIMIT ? Full query	1,930 0.90/sec 1.00/call 0.01%	456,792.41 ms 212.416 ms/sec 236.680 ms/call 1.35%	114,105 53.06/sec 59.12/call 0.06%	244,412,359 blks 113.66K blks/sec 126.64K blks/call 17.36%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1,747 blks 0.81 blks/sec 0.91 blks/call 0.13%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	UPDATE "ci_builds" SET "status" = ?, "updated_at" = ?, "lock_version" = ? WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."lock_version" = ? OR "ci_builds"."lock_version" IS NULL) AND "ci_builds"."id" = ? Full query	75,793 35.24/sec 1.00/call 0.23%	411,263.09 ms 191.244 ms/sec 5.426 ms/call 1.22%	11,004 5.12/sec 0.15/call 0.01%	1,410,557 blks 655.93 blks/sec 18.61 blks/call 0.10%	4,860 blks 2.26 blks/sec 0.06 blks/call 0.58%	16,970 blks 7.89 blks/sec 0.22 blks/call 1.27%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2,068.68 ms 0.962 ms/sec 0.027 ms/call 0.58%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	UPDATE "project_mirror_data" SET "status" = ?, "last_update_scheduled_at" = ? WHERE "project_mirror_data"."id" = ? Full query	109,184 50.77/sec 1.00/call 0.33%	399,211.11 ms 185.640 ms/sec 3.656 ms/call 1.18%	109,184 50.77/sec 1.00/call 0.06%	3,679,062 blks 1.72K blks/sec 33.70 blks/call 0.26%	24 blks 0.01 blks/sec 0.00 blks/call 0.00%	238,817 blks 111.05 blks/sec 2.19 blks/call 17.82%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "members"."source_type" = ? AND "namespaces"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND (members.access_level >= ?) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces... Full query	26 0.01/sec 1.00/call 0.00%	296,454.52 ms 137.856 ms/sec 11402.097 ms/call 0.88%	26 0.01/sec 1.00/call 0.00%	248,296,143 blks 115.47K blks/sec 9.55M blks/call 17.63%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	11,543 blks 5.37 blks/sec 443.96 blks/call 0.86%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	UPDATE "project_mirror_data" SET "status" = ?, "last_update_started_at" = ? WHERE "project_mirror_data"."id" = ? Full query	108,044 50.24/sec 1.00/call 0.33%	292,003.90 ms 135.787 ms/sec 2.703 ms/call 0.86%	108,044 50.24/sec 1.00/call 0.06%	4,081,204 blks 1.90K blks/sec 37.77 blks/call 0.29%	16 blks 0.01 blks/sec 0.00 blks/call 0.00%	2,031 blks 0.94 blks/sec 0.02 blks/call 0.15%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT (CASE WHEN (SELECT count() FROM "ci_builds" WHERE "ci_builds"."stage_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_failure = ? OR status IN (?,?,?,?,?,?,?)))=( SELECT count() FROM "ci_builds" WHERE "ci_builds"."stage_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_failure = ? OR status IN (?,?,?,?,?,?,?)) AND ("ci_builds"."status" IN (?))) AND (SELECT count() > ? FROM "ci_builds" WHERE "ci_builds"."stage_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND "ci_builds"."allow_failure" = ? AND "ci_builds"."status" IN (?, ?)) THEN ? WHEN ( SELECT count() FROM "ci_builds" WHERE "ci_builds"."stage_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_failure = ? OR status IN (?,?,?,?,?,?,?)))=(SELECT count( *) FROM "ci_builds" WHERE "ci_builds"."stage_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_fa... Full query	104,559 48.62/sec 1.00/call 0.32%	283,237.54 ms 131.710 ms/sec 2.709 ms/call 0.84%	1,631,987 758.90/sec 15.61/call 0.91%	108,877,621 blks 50.63K blks/sec 1.05K blks/call 7.73%	676 blks 0.31 blks/sec 0.01 blks/call 0.08%	764 blks 0.36 blks/sec 0.01 blks/call 0.06%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	467.29 ms 0.217 ms/sec 0.004 ms/call 0.13%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT "web_hooks".* FROM "web_hooks" WHERE "web_hooks"."type" IN (?) AND "web_hooks"."group_id" = ? AND "web_hooks"."job_events" = ? Full query	42,403 19.72/sec 1.00/call 0.13%	228,849.19 ms 106.419 ms/sec 5.397 ms/call 0.68%	1,060 0.49/sec 0.02/call 0.00%	115,590,578 blks 53.76K blks/sec 2.73K blks/call 8.21%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT "users".* FROM "users" WHERE "users"."unlock_token" = ? ORDER BY "users"."id" ASC LIMIT ? Full query	22 0.01/sec 1.00/call 0.00%	208,875.30 ms 97.130 ms/sec 9494.332 ms/call 0.62%	0 0.00/sec 0.00/call 0.00%	98,786,043 blks 45.94K blks/sec 4.50M blks/call 7.02%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	13,730 blks 6.38 blks/sec 624.09 blks/call 1.02%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT ? AS one FROM ((SELECT "projects".* FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" IN (SELECT "elasticsearch_indexed_namespaces"."namespace_id" FROM "elasticsearch_indexed_namespaces")) UNION ( SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" FROM "base_and_descendants" AS "namespaces")) UNION (SELECT "projects".* FROM "projects" WHERE "projects"."id" IN (SELECT "elasticsearch_indexed_projects"."project_id" FROM "elasticsearch_indexed_projects"))) projects WHERE "projects"."id" = ? LIMIT ? Full query	183,557 85.36/sec 1.00/call 0.55%	136,712.12 ms 63.573 ms/sec 0.745 ms/call 0.40%	2,524 1.17/sec 0.01/call 0.00%	52,927,303 blks 24.62K blks/sec 288.34 blks/call 3.76%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2,477 blks 1.15 blks/sec 0.01 blks/call 0.18%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	WITH RECURSIVE "namespaces_cte" AS (SELECT "namespaces"."id", "members"."access_level" FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "members"."source_type" = ? AND "namespaces"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL UNION SELECT "namespaces"."id", GREATEST("members"."access_level", "namespaces_cte"."access_level") AS access_level FROM "namespaces" INNER JOIN "namespaces_cte" ON "namespaces_cte"."id" = "namespaces"."parent_id" LEFT OUTER JOIN "members" ON "members"."source_id" = "namespaces"."id" AND "members"."source_type" = ? AND "members"."requested_at" IS NULL AND "members"."user_id" = ? WHERE "namespaces"."type" IN (?)) SELECT "project_authorizations"."project_id", MAX(access_level) AS access_level FROM (SELECT projects.id AS project_id, members.access_level FROM "projects" INNER JOIN "members" ON "projects"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND... Full query	11,624 5.41/sec 1.00/call 0.04%	131,962.25 ms 61.365 ms/sec 11.353 ms/call 0.39%	20,179,217 9.39K/sec 1.74K/call 11.27%	55,125,102 blks 25.64K blks/sec 4.75K blks/call 3.91%	2,329 blks 1.08 blks/sec 0.20 blks/call 0.28%	469 blks 0.22 blks/sec 0.04 blks/call 0.03%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	981.42 ms 0.456 ms/sec 0.084 ms/call 0.28%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	UPDATE "project_mirror_data" SET "jid" = ? WHERE "project_mirror_data"."id" = ? Full query	109,179 50.77/sec 1.00/call 0.33%	128,780.77 ms 59.885 ms/sec 1.180 ms/call 0.38%	109,179 50.77/sec 1.00/call 0.06%	3,969,153 blks 1.85K blks/sec 36.35 blks/call 0.28%	33 blks 0.02 blks/sec 0.00 blks/call 0.00%	61,899 blks 28.78 blks/sec 0.57 blks/call 4.62%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.05 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT (CASE WHEN (SELECT count() FROM "ci_builds" WHERE "ci_builds"."commit_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_failure = ? OR status IN (?,?,?,?,?,?,?)))=( SELECT count() FROM "ci_builds" WHERE "ci_builds"."commit_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_failure = ? OR status IN (?,?,?,?,?,?,?)) AND ("ci_builds"."status" IN (?))) AND (SELECT count() > ? FROM "ci_builds" WHERE "ci_builds"."commit_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND "ci_builds"."allow_failure" = ? AND "ci_builds"."status" IN (?, ?)) THEN ? WHEN ( SELECT count() FROM "ci_builds" WHERE "ci_builds"."commit_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (allow_failure = ? OR status IN (?,?,?,?,?,?,?)))=(SELECT count( *) FROM "ci_builds" WHERE "ci_builds"."commit_id" = ? AND ("ci_builds"."retried" = ? OR "ci_builds"."retried" IS NULL) AND (all... Full query	126,435 58.79/sec 1.00/call 0.38%	91,800.13 ms 42.689 ms/sec 0.726 ms/call 0.27%	2,678,944 1.25K/sec 21.19/call 1.50%	58,327,240 blks 27.13K blks/sec 461.32 blks/call 4.14%	1,138 blks 0.53 blks/sec 0.01 blks/call 0.14%	750 blks 0.35 blks/sec 0.01 blks/call 0.06%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	883.19 ms 0.411 ms/sec 0.007 ms/call 0.25%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr( d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, c.collname, col_description( a.attrelid, a.attnum) AS comment FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum LEFT JOIN pg_type t ON a.atttypid = t.oid LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation WHERE a.attrelid = ?::regclass AND a.attnum > ? AND NOT a.attisdropped ORDER BY a.attnum Full query	140,047 65.12/sec 1.00/call 0.42%	87,199.20 ms 40.549 ms/sec 0.623 ms/call 0.26%	1,804,015 838.90/sec 12.88/call 1.01%	10,264,423 blks 4.78K blks/sec 73.29 blks/call 0.73%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	UPDATE "project_mirror_data" SET "status" = ?, "next_execution_timestamp" = ?, "last_update_at" = ?, "last_successful_update_at" = ? WHERE "project_mirror_data"."id" = ? Full query	107,716 50.09/sec 1.00/call 0.33%	83,660.06 ms 38.903 ms/sec 0.777 ms/call 0.25%	107,716 50.09/sec 1.00/call 0.06%	4,307,564 blks 2.01K blks/sec 39.99 blks/call 0.31%	29 blks 0.01 blks/sec 0.00 blks/call 0.00%	4,094 blks 1.90 blks/sec 0.04 blks/call 0.31%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.01 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT ? AS one FROM ((WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces".* FROM ((SELECT "namespaces".* FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "members"."source_type" = ? AND "namespaces"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL) UNION (SELECT namespaces.* FROM "projects" INNER JOIN "namespaces" ON "namespaces"."id" = "projects"."namespace_id" INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id" WHERE "project_authorizations"."user_id" = ?)) namespaces WHERE "namespaces"."type" IN (?)) UNION (SELECT "namespaces".* FROM "namespaces", "base_and_ancestors" WHERE "namespaces"."type" IN (?) AND "namespaces"."id" = "base_and_ancestors"."parent_id")), "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "member... Full query	55 0.03/sec 1.00/call 0.00%	74,694.06 ms 34.734 ms/sec 1358.074 ms/call 0.22%	55 0.03/sec 1.00/call 0.00%	25,076,970 blks 11.67K blks/sec 455.95K blks/call 1.78%	12 blks 0.01 blks/sec 0.22 blks/call 0.00%	126 blks 0.06 blks/sec 2.29 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	7.97 ms 0.004 ms/sec 0.145 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = ? AND "routes"."source_type" = ? LIMIT ? Full query	1,341,655 623.89/sec 1.00/call 4.05%	62,811.08 ms 29.208 ms/sec 0.047 ms/call 0.19%	1,341,655 623.89/sec 1.00/call 0.75%	6,711,351 blks 3.13K blks/sec 5.00 blks/call 0.48%	2,933 blks 1.36 blks/sec 0.00 blks/call 0.35%	169 blks 0.08 blks/sec 0.00 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	508.56 ms 0.236 ms/sec 0.000 ms/call 0.14%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT "projects".* FROM "projects" WHERE "projects"."id" = ? LIMIT ? Full query	1,205,325 560.50/sec 1.00/call 3.64%	62,451.45 ms 29.041 ms/sec 0.052 ms/call 0.18%	1,205,254 560.46/sec 1.00/call 0.67%	4,965,791 blks 2.31K blks/sec 4.12 blks/call 0.35%	1 blks 0.00 blks/sec 0.00 blks/call 0.00%	157 blks 0.07 blks/sec 0.00 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.02 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica servers:

Replica (`10.220.16.101`)

Nothing found

Replica (`10.220.16.102`)

Start: 2019-11-11T13:05:32.574522+00:00
End: 2019-11-11T13:36:01.607707+00:00
Period seconds: 1829.03319
Period age: 00:30:29.033185

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = 'Namespace' AND "namespaces"."type" IN ('Group') AND "members"."user_id" = 4589059 AND "members"."requested_at" IS NULL AND (members.access_level >= 30) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND (EXISTS (SELECT 1 FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN ('silver', ... Full query	49 0.03/sec 1.00/call 0.00%	489,345.37 ms 267.543 ms/sec 9986.640 ms/call 17.34%	49 0.03/sec 1.00/call 0.00%	409,326,597 blks 223.80K blks/sec 8.36M blks/call 33.09%	61 blks 0.03 blks/sec 1.24 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.72 ms 0.000 ms/sec 0.015 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id))) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	32 0.02/sec 1.00/call 0.00%	232,126.86 ms 126.912 ms/sec 7253.964 ms/call 8.22%	1 0.00/sec 0.03/call 0.00%	97,534,114 blks 53.33K blks/sec 3.05M blks/call 7.88%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	47 0.03/sec 1.00/call 0.00%	199,176.18 ms 108.897 ms/sec 4237.791 ms/call 7.06%	47 0.03/sec 1.00/call 0.00%	18,157,127 blks 9.93K blks/sec 386.33K blks/call 1.47%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	431 0.24/sec 1.00/call 0.00%	198,777.75 ms 108.679 ms/sec 461.201 ms/call 7.04%	66,902 36.58/sec 155.23/call 0.19%	136,308,119 blks 74.53K blks/sec 316.27K blks/call 11.02%	144 blks 0.08 blks/sec 0.33 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	3.49 ms 0.002 ms/sec 0.008 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND ("ci_builds"."status" IN ('running')) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = 1) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND ("ci_builds"."status" IN ('pending')) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = TRUE AND "projects"."pending_delete" = FALSE AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) AND (projects.visibility_level=20 OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHER... Full query	441 0.24/sec 1.00/call 0.00%	194,568.98 ms 106.378 ms/sec 441.200 ms/call 6.89%	65,331 35.72/sec 148.14/call 0.19%	138,873,227 blks 75.93K blks/sec 314.91K blks/call 11.23%	115 blks 0.06 blks/sec 0.26 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1.84 ms 0.001 ms/sec 0.004 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	10 0.01/sec 1.00/call 0.00%	78,382.50 ms 42.855 ms/sec 7838.250 ms/call 2.78%	210 0.11/sec 21.00/call 0.00%	3,863,648 blks 2.12K blks/sec 386.37K blks/call 0.31%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	175 0.10/sec 1.00/call 0.00%	68,531.85 ms 37.469 ms/sec 391.611 ms/call 2.43%	9,281 5.07/sec 53.03/call 0.03%	46,624,627 blks 25.50K blks/sec 266.43K blks/call 3.77%	22 blks 0.01 blks/sec 0.13 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	10.02 ms 0.005 ms/sec 0.057 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT COUNT(*) FROM (SELECT ? AS one FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? LIMIT ?) subquery_for_count Full query	54 0.03/sec 1.00/call 0.00%	60,889.19 ms 33.290 ms/sec 1127.578 ms/call 2.16%	54 0.03/sec 1.00/call 0.00%	23,405,123 blks 12.80K blks/sec 433.43K blks/call 1.89%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	101 0.06/sec 1.00/call 0.00%	53,173.48 ms 29.072 ms/sec 526.470 ms/call 1.88%	10,194 5.57/sec 100.93/call 0.03%	16,745,821 blks 9.16K blks/sec 165.81K blks/call 1.35%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("issues"."title" ILIKE ? OR "issues"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	12 0.01/sec 1.00/call 0.00%	50,698.86 ms 27.719 ms/sec 4224.905 ms/call 1.80%	12 0.01/sec 1.00/call 0.00%	900,130 blks 492.13 blks/sec 75.02K blks/call 0.07%	41,193 blks 22.52 blks/sec 3.44K blks/call 2.81%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	605.71 ms 0.331 ms/sec 50.475 ms/call 0.85%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = 10463077 ORDER BY "environments"."id" ASC LIMIT 101 OFFSET 0 Full query	419 0.23/sec 1.00/call 0.00%	46,389.04 ms 25.363 ms/sec 110.714 ms/call 1.64%	28,600 15.64/sec 68.26/call 0.08%	44,208,408 blks 24.18K blks/sec 105.51K blks/call 3.57%	2 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.04 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	6,448 3.53/sec 1.00/call 0.07%	45,775.76 ms 25.027 ms/sec 7.099 ms/call 1.62%	1,520 0.83/sec 0.24/call 0.00%	19,807,104 blks 10.83K blks/sec 3.08K blks/call 1.60%	17,072 blks 9.33 blks/sec 2.65 blks/call 1.16%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	330.02 ms 0.180 ms/sec 0.051 ms/call 0.46%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	722,218 394.86/sec 1.00/call 7.38%	45,773.81 ms 25.026 ms/sec 0.063 ms/call 1.62%	711,946 389.25/sec 0.99/call 2.02%	6,461,332 blks 3.54K blks/sec 8.95 blks/call 0.52%	9,892 blks 5.41 blks/sec 0.01 blks/call 0.67%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	760.55 ms 0.416 ms/sec 0.001 ms/call 1.07%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	899 0.49/sec 1.00/call 0.01%	45,612.06 ms 24.938 ms/sec 50.736 ms/call 1.62%	899 0.49/sec 1.00/call 0.00%	3,422,493 blks 1.88K blks/sec 3.81K blks/call 0.28%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = ? AND "deployments"."status" = ? ORDER BY deployments.id DESC LIMIT ? Full query	11,949 6.53/sec 1.00/call 0.12%	45,074.61 ms 24.644 ms/sec 3.772 ms/call 1.60%	9,633 5.27/sec 0.81/call 0.03%	15,373,654 blks 8.41K blks/sec 1.29K blks/call 1.24%	1,133 blks 0.62 blks/sec 0.09 blks/call 0.08%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	30.88 ms 0.017 ms/sec 0.003 ms/call 0.04%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT COUNT(*) FROM (SELECT ? AS one FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."pending_delete" = ? LIMIT ?) subquery_for_count Full query	3 0.00/sec 1.00/call 0.00%	43,486.87 ms 23.776 ms/sec 14495.622 ms/call 1.54%	3 0.00/sec 1.00/call 0.00%	17,167,635 blks 9.39K blks/sec 5.73M blks/call 1.39%	1 blks 0.00 blks/sec 0.33 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.01 ms 0.000 ms/sec 0.004 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE ( issues.confidential IS NOT TRUE OR (issues.confidential = ? AND (issues.author_id = ? OR EXISTS (SELECT ? FROM issue_assignees WHERE user_id = ? AND issue_id = issues.id) OR EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = issues.project_id) AND (project_authorizations.access_level >= ?))))) AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_l... Full query	8 0.00/sec 1.00/call 0.00%	41,865.54 ms 22.889 ms/sec 5233.192 ms/call 1.48%	8 0.00/sec 1.00/call 0.00%	730,470 blks 399.37 blks/sec 91.31K blks/call 0.06%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT "users".* FROM "users" WHERE ("users"."state" IN (?)) AND (ghost IS NOT TRUE) AND "users"."bot_type" IS NULL AND (("users"."name" ILIKE ? OR "users"."username" ILIKE ?) OR "users"."email" = ?) ORDER BY CASE WHEN users.name = ? THEN ? WHEN users.username = ? THEN ? WHEN users.email = ? THEN ? ELSE ? END , "users"."name" ASC LIMIT ? Full query	404 0.22/sec 1.00/call 0.00%	34,894.81 ms 19.078 ms/sec 86.373 ms/call 1.24%	3,804 2.08/sec 9.42/call 0.01%	3,231,170 blks 1.77K blks/sec 8.00K blks/call 0.26%	2,512 blks 1.37 blks/sec 6.22 blks/call 0.17%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	24.01 ms 0.013 ms/sec 0.059 ms/call 0.03%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT projects.id FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	5 0.00/sec 1.00/call 0.00%	31,730.52 ms 17.348 ms/sec 6346.104 ms/call 1.12%	0 0.00/sec 0.00/call 0.00%	1,931,903 blks 1.06K blks/sec 386.39K blks/call 0.16%	1 blks 0.00 blks/sec 0.20 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.03 ms 0.000 ms/sec 0.005 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT projects.id FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	51 0.03/sec 1.00/call 0.00%	28,042.77 ms 15.332 ms/sec 549.858 ms/call 0.99%	2 0.00/sec 0.04/call 0.00%	11,238,240 blks 6.15K blks/sec 220.36K blks/call 0.91%	8 blks 0.00 blks/sec 0.16 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.08 ms 0.000 ms/sec 0.002 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.103`)

Start: 2019-11-11T13:05:33.920039+00:00
End: 2019-11-11T13:36:46.045187+00:00
Period seconds: 1872.12515
Period age: 00:31:12.125148

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = 'Namespace' AND "namespaces"."type" IN ('Group') AND "members"."user_id" = 4600163 AND "members"."requested_at" IS NULL AND (members.access_level >= 30) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND (EXISTS (SELECT 1 FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN ('silver', ... Full query	57 0.03/sec 1.00/call 0.00%	606,530.82 ms 323.980 ms/sec 10640.892 ms/call 22.38%	57 0.03/sec 1.00/call 0.00%	465,437,547 blks 248.62K blks/sec 8.17M blks/call 38.84%	211 blks 0.11 blks/sec 3.70 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	11.77 ms 0.006 ms/sec 0.207 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id))) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	29 0.02/sec 1.00/call 0.00%	229,192.71 ms 122.424 ms/sec 7903.197 ms/call 8.46%	0 0.00/sec 0.00/call 0.00%	90,683,556 blks 48.44K blks/sec 3.13M blks/call 7.57%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	449 0.24/sec 1.00/call 0.00%	226,969.39 ms 121.236 ms/sec 505.500 ms/call 8.37%	56,416 30.13/sec 125.65/call 0.16%	141,781,728 blks 75.74K blks/sec 315.78K blks/call 11.83%	112 blks 0.06 blks/sec 0.25 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	5.12 ms 0.003 ms/sec 0.011 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	448 0.24/sec 1.00/call 0.00%	220,663.66 ms 117.868 ms/sec 492.553 ms/call 8.14%	61,957 33.09/sec 138.30/call 0.18%	141,798,821 blks 75.75K blks/sec 316.52K blks/call 11.83%	140 blks 0.07 blks/sec 0.31 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	17.93 ms 0.010 ms/sec 0.040 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	31 0.02/sec 1.00/call 0.00%	137,135.07 ms 73.251 ms/sec 4423.712 ms/call 5.06%	31 0.02/sec 1.00/call 0.00%	11,975,964 blks 6.40K blks/sec 386.33K blks/call 1.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	192 0.10/sec 1.00/call 0.00%	80,260.43 ms 42.871 ms/sec 418.023 ms/call 2.96%	8,956 4.78/sec 46.65/call 0.03%	50,836,701 blks 27.16K blks/sec 264.78K blks/call 4.24%	27 blks 0.01 blks/sec 0.14 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	17.72 ms 0.009 ms/sec 0.092 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	6,816 3.64/sec 1.00/call 0.07%	53,276.78 ms 28.458 ms/sec 7.816 ms/call 1.97%	1,638 0.87/sec 0.24/call 0.00%	20,676,485 blks 11.05K blks/sec 3.04K blks/call 1.73%	19,735 blks 10.54 blks/sec 2.90 blks/call 1.23%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	823.29 ms 0.440 ms/sec 0.121 ms/call 0.58%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	739,414 394.96/sec 1.00/call 7.65%	51,023.24 ms 27.254 ms/sec 0.069 ms/call 1.88%	728,850 389.32/sec 0.99/call 2.11%	6,615,489 blks 3.54K blks/sec 8.95 blks/call 0.55%	8,955 blks 4.78 blks/sec 0.01 blks/call 0.56%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	566.34 ms 0.303 ms/sec 0.001 ms/call 0.40%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	6 0.00/sec 1.00/call 0.00%	48,099.77 ms 25.693 ms/sec 8016.628 ms/call 1.77%	126 0.07/sec 21.00/call 0.00%	2,318,058 blks 1.24K blks/sec 386.35K blks/call 0.19%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	875 0.47/sec 1.00/call 0.01%	45,688.31 ms 24.405 ms/sec 52.215 ms/call 1.69%	875 0.47/sec 1.00/call 0.00%	3,331,125 blks 1.78K blks/sec 3.81K blks/call 0.28%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? ORDER BY "environments"."id" ASC LIMIT ? OFFSET ? Full query	425 0.23/sec 1.00/call 0.00%	40,906.99 ms 21.851 ms/sec 96.252 ms/call 1.51%	28,983 15.48/sec 68.20/call 0.08%	35,074,655 blks 18.74K blks/sec 82.53K blks/call 2.93%	7 blks 0.00 blks/sec 0.02 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.10 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	68 0.04/sec 1.00/call 0.00%	40,771.96 ms 21.778 ms/sec 599.588 ms/call 1.50%	6,868 3.67/sec 101.00/call 0.02%	10,333,726 blks 5.52K blks/sec 151.97K blks/call 0.86%	3,215 blks 1.72 blks/sec 47.28 blks/call 0.20%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	96.10 ms 0.051 ms/sec 1.413 ms/call 0.07%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("issues"."title" ILIKE ? OR "issues"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	9 0.00/sec 1.00/call 0.00%	39,891.44 ms 21.308 ms/sec 4432.383 ms/call 1.47%	9 0.00/sec 1.00/call 0.00%	715,518 blks 382.20 blks/sec 79.51K blks/call 0.06%	15,570 blks 8.32 blks/sec 1.73K blks/call 0.97%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	576.62 ms 0.308 ms/sec 64.069 ms/call 0.41%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT "users".* FROM "users" WHERE ("users"."state" IN (?)) AND (ghost IS NOT TRUE) AND "users"."bot_type" IS NULL AND (("users"."name" ILIKE ? OR "users"."username" ILIKE ?) OR "users"."email" = ?) ORDER BY CASE WHEN users.name = ? THEN ? WHEN users.username = ? THEN ? WHEN users.email = ? THEN ? ELSE ? END , "users"."name" ASC LIMIT ? Full query	400 0.21/sec 1.00/call 0.00%	34,240.95 ms 18.290 ms/sec 85.602 ms/call 1.26%	3,692 1.97/sec 9.23/call 0.01%	3,142,484 blks 1.68K blks/sec 7.86K blks/call 0.26%	1,895 blks 1.01 blks/sec 4.74 blks/call 0.12%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	69.12 ms 0.037 ms/sec 0.173 ms/call 0.05%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() ; Full query	124 0.07/sec 1.00/call 0.00%	30,421.11 ms 16.250 ms/sec 245.332 ms/call 1.12%	124 0.07/sec 1.00/call 0.00%	18,260,187 blks 9.76K blks/sec 147.26K blks/call 1.52%	2 blks 0.00 blks/sec 0.02 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.03 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = ? ORDER BY "ci_pipelines"."id" DESC LIMIT ? OFFSET ? Full query	3,938 2.10/sec 1.00/call 0.04%	30,373.21 ms 16.224 ms/sec 7.713 ms/call 1.12%	73,788 39.41/sec 18.74/call 0.21%	5,368,316 blks 2.87K blks/sec 1.37K blks/call 0.45%	388,673 blks 207.61 blks/sec 98.70 blks/call 24.20%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	9,712.16 ms 5.188 ms/sec 2.466 ms/call 6.85%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = ? AND "deployments"."status" = ? ORDER BY deployments.id DESC LIMIT ? Full query	10,415 5.56/sec 1.00/call 0.11%	28,874.12 ms 15.423 ms/sec 2.772 ms/call 1.07%	8,252 4.41/sec 0.79/call 0.02%	8,937,460 blks 4.78K blks/sec 858.13 blks/call 0.75%	1,176 blks 0.63 blks/sec 0.11 blks/call 0.07%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	98.52 ms 0.053 ms/sec 0.009 ms/call 0.07%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() - ?::interval ; Full query	124 0.07/sec 1.00/call 0.00%	28,821.13 ms 15.395 ms/sec 232.428 ms/call 1.06%	124 0.07/sec 1.00/call 0.00%	18,080,000 blks 9.66K blks/sec 145.81K blks/call 1.51%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT COUNT(DISTINCT "events"."id") FROM "events" INNER JOIN "projects" ON "projects"."id" = "events"."project_id" LEFT OUTER JOIN "project_features" ON "project_features"."project_id" = "projects"."id" LEFT OUTER JOIN "project_import_data" ON "project_import_data"."project_id" = "projects"."id" LEFT OUTER JOIN "namespaces" ON "namespaces"."id" = "projects"."namespace_id" LEFT OUTER JOIN "users" ON "users"."id" = "events"."author_id" WHERE "events"."project_id" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	2,680 1.43/sec 1.00/call 0.03%	26,836.88 ms 14.335 ms/sec 10.014 ms/call 0.99%	2,680 1.43/sec 1.00/call 0.01%	3,917,506 blks 2.10K blks/sec 1.47K blks/call 0.33%	122,191 blks 65.27 blks/sec 45.59 blks/call 7.61%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	10,685.35 ms 5.708 ms/sec 3.987 ms/call 7.54%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND (project_mirror_data.next_execution_timestamp - project_mirror_data.last_update_at) <= ?::interval AND project_mirror_data.last_update_at < NOW() - ?::interval ; Full query	124 0.07/sec 1.00/call 0.00%	25,252.33 ms 13.489 ms/sec 203.648 ms/call 0.93%	124 0.07/sec 1.00/call 0.00%	13,941,849 blks 7.45K blks/sec 112.44K blks/call 1.16%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.104`)

Start: 2019-11-11T13:05:35.26518+00:00
End: 2019-11-11T13:37:44.522444+00:00
Period seconds: 1929.25726
Period age: 00:32:09.257264

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "members"."source_type" = ? AND "namespaces"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND (members.access_level >= ?) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces... Full query	58 0.03/sec 1.00/call 0.00%	584,838.06 ms 303.142 ms/sec 10083.415 ms/call 22.84%	58 0.03/sec 1.00/call 0.00%	487,069,042 blks 252.47K blks/sec 8.40M blks/call 39.71%	109 blks 0.06 blks/sec 1.88 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	3.58 ms 0.002 ms/sec 0.062 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	473 0.25/sec 1.00/call 0.00%	220,005.97 ms 114.037 ms/sec 465.129 ms/call 8.59%	61,219 31.73/sec 129.43/call 0.16%	149,919,506 blks 77.71K blks/sec 316.96K blks/call 12.22%	119 blks 0.06 blks/sec 0.25 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	7.18 ms 0.004 ms/sec 0.015 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	471 0.24/sec 1.00/call 0.00%	214,362.45 ms 111.111 ms/sec 455.122 ms/call 8.37%	58,416 30.28/sec 124.03/call 0.16%	148,764,324 blks 77.11K blks/sec 315.85K blks/call 12.13%	163 blks 0.08 blks/sec 0.35 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	19.98 ms 0.010 ms/sec 0.042 ms/call 0.02%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id))) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	23 0.01/sec 1.00/call 0.00%	171,129.76 ms 88.702 ms/sec 7440.424 ms/call 6.68%	0 0.00/sec 0.00/call 0.00%	71,920,137 blks 37.28K blks/sec 3.13M blks/call 5.86%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 4465626 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) Full query	24 0.01/sec 1.00/call 0.00%	106,859.34 ms 55.389 ms/sec 4452.472 ms/call 4.17%	24 0.01/sec 1.00/call 0.00%	9,271,748 blks 4.81K blks/sec 386.33K blks/call 0.76%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND ("ci_builds"."status" IN ('running')) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = 1) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND ("ci_builds"."status" IN ('pending')) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = TRUE AND "projects"."pending_delete" = FALSE AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) AND (projects.visibility_level=20 OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHER... Full query	197 0.10/sec 1.00/call 0.00%	80,949.99 ms 41.959 ms/sec 410.914 ms/call 3.16%	9,488 4.92/sec 48.16/call 0.03%	52,784,571 blks 27.37K blks/sec 267.95K blks/call 4.30%	26 blks 0.01 blks/sec 0.13 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	17.55 ms 0.009 ms/sec 0.089 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? ORDER BY "environments"."id" ASC LIMIT ? OFFSET ? Full query	483 0.25/sec 1.00/call 0.00%	49,791.63 ms 25.809 ms/sec 103.088 ms/call 1.94%	35,193 18.24/sec 72.86/call 0.09%	44,658,574 blks 23.15K blks/sec 92.47K blks/call 3.64%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	6,976 3.62/sec 1.00/call 0.07%	49,061.04 ms 25.430 ms/sec 7.033 ms/call 1.92%	1,676 0.87/sec 0.24/call 0.00%	21,183,186 blks 10.98K blks/sec 3.04K blks/call 1.73%	14,450 blks 7.49 blks/sec 2.07 blks/call 0.84%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	856.42 ms 0.444 ms/sec 0.123 ms/call 0.67%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	761,611 394.77/sec 1.00/call 7.58%	48,612.14 ms 25.197 ms/sec 0.064 ms/call 1.90%	750,854 389.19/sec 0.99/call 2.00%	6,814,261 blks 3.54K blks/sec 8.95 blks/call 0.56%	9,112 blks 4.72 blks/sec 0.01 blks/call 0.53%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	552.24 ms 0.286 ms/sec 0.001 ms/call 0.43%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("issues"."title" ILIKE ? OR "issues"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	10 0.01/sec 1.00/call 0.00%	47,496.70 ms 24.619 ms/sec 4749.670 ms/call 1.85%	10 0.01/sec 1.00/call 0.00%	742,787 blks 385.01 blks/sec 74.28K blks/call 0.06%	18,935 blks 9.81 blks/sec 1.90K blks/call 1.10%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	514.48 ms 0.267 ms/sec 51.448 ms/call 0.40%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	873 0.45/sec 1.00/call 0.01%	45,738.08 ms 23.708 ms/sec 52.392 ms/call 1.79%	873 0.45/sec 1.00/call 0.00%	3,323,511 blks 1.73K blks/sec 3.81K blks/call 0.27%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = ? AND "deployments"."status" = ? ORDER BY deployments.id DESC LIMIT ? Full query	11,906 6.17/sec 1.00/call 0.12%	40,096.49 ms 20.783 ms/sec 3.368 ms/call 1.57%	9,580 4.97/sec 0.80/call 0.03%	12,975,118 blks 6.73K blks/sec 1.09K blks/call 1.06%	2,275 blks 1.18 blks/sec 0.19 blks/call 0.13%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	169.48 ms 0.088 ms/sec 0.014 ms/call 0.13%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 3308413 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) AND "projects"."pending_delete" = FALSE ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT 51 OFFSET 150 Full query	4 0.00/sec 1.00/call 0.00%	32,571.93 ms 16.883 ms/sec 8142.984 ms/call 1.27%	84 0.04/sec 21.00/call 0.00%	1,545,425 blks 801.05 blks/sec 386.36K blks/call 0.13%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT "users".* FROM "users" WHERE ("users"."state" IN (?)) AND (ghost IS NOT TRUE) AND "users"."bot_type" IS NULL AND (("users"."name" ILIKE ? OR "users"."username" ILIKE ?) OR "users"."email" = ?) ORDER BY CASE WHEN users.name = ? THEN ? WHEN users.username = ? THEN ? WHEN users.email = ? THEN ? ELSE ? END , "users"."name" ASC LIMIT ? Full query	455 0.24/sec 1.00/call 0.00%	31,867.64 ms 16.518 ms/sec 70.039 ms/call 1.24%	4,288 2.22/sec 9.42/call 0.01%	3,075,835 blks 1.60K blks/sec 6.77K blks/call 0.25%	1,909 blks 0.99 blks/sec 4.20 blks/call 0.11%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	47.78 ms 0.025 ms/sec 0.105 ms/call 0.04%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() ; Full query	130 0.07/sec 1.00/call 0.00%	31,672.79 ms 16.417 ms/sec 243.637 ms/call 1.24%	130 0.07/sec 1.00/call 0.00%	19,035,253 blks 9.87K blks/sec 146.43K blks/call 1.55%	2 blks 0.00 blks/sec 0.02 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.84 ms 0.000 ms/sec 0.006 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = ? ORDER BY "ci_pipelines"."id" DESC LIMIT ? OFFSET ? Full query	3,884 2.01/sec 1.00/call 0.04%	31,031.94 ms 16.085 ms/sec 7.990 ms/call 1.21%	75,464 39.12/sec 19.43/call 0.20%	5,877,933 blks 3.05K blks/sec 1.52K blks/call 0.48%	366,519 blks 189.98 blks/sec 94.37 blks/call 21.28%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	10,503.83 ms 5.444 ms/sec 2.704 ms/call 8.24%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() - ?::interval ; Full query	130 0.07/sec 1.00/call 0.00%	29,838.20 ms 15.466 ms/sec 229.525 ms/call 1.17%	130 0.07/sec 1.00/call 0.00%	18,903,194 blks 9.80K blks/sec 145.41K blks/call 1.54%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND (project_mirror_data.next_execution_timestamp - project_mirror_data.last_update_at) <= ?::interval AND project_mirror_data.last_update_at < NOW() - ?::interval ; Full query	130 0.07/sec 1.00/call 0.00%	25,037.89 ms 12.978 ms/sec 192.599 ms/call 0.98%	130 0.07/sec 1.00/call 0.00%	14,586,769 blks 7.57K blks/sec 112.21K blks/call 1.19%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT COUNT(DISTINCT "events"."id") FROM "events" INNER JOIN "projects" ON "projects"."id" = "events"."project_id" LEFT OUTER JOIN "project_features" ON "project_features"."project_id" = "projects"."id" LEFT OUTER JOIN "project_import_data" ON "project_import_data"."project_id" = "projects"."id" LEFT OUTER JOIN "namespaces" ON "namespaces"."id" = "projects"."namespace_id" LEFT OUTER JOIN "users" ON "users"."id" = "events"."author_id" WHERE "events"."project_id" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	2,656 1.38/sec 1.00/call 0.03%	24,639.89 ms 12.772 ms/sec 9.277 ms/call 0.96%	2,656 1.38/sec 1.00/call 0.01%	4,072,569 blks 2.12K blks/sec 1.54K blks/call 0.33%	278,547 blks 144.38 blks/sec 104.87 blks/call 16.17%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	7,645.61 ms 3.963 ms/sec 2.879 ms/call 6.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT "users".* FROM "users" WHERE "users"."id" = ? LIMIT ? Full query	570,360 295.64/sec 1.00/call 5.68%	23,574.50 ms 12.219 ms/sec 0.041 ms/call 0.92%	570,067 295.49/sec 1.00/call 1.52%	2,281,134 blks 1.19K blks/sec 4.00 blks/call 0.19%	6 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.10 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.105`)

Start: 2019-11-11T13:05:36.541154+00:00
End: 2019-11-11T13:38:58.630579+00:00
Period seconds: 2002.08943
Period age: 00:33:22.089425

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = 'Namespace' AND "namespaces"."type" IN ('Group') AND "members"."user_id" = 933598 AND "members"."requested_at" IS NULL AND (members.access_level >= 30) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND (EXISTS (SELECT 1 FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN ('silver', '... Full query	60 0.03/sec 1.00/call 0.00%	605,301.63 ms 302.335 ms/sec 10088.360 ms/call 22.44%	60 0.03/sec 1.00/call 0.00%	551,664,109 blks 275.55K blks/sec 9.20M blks/call 40.61%	66 blks 0.03 blks/sec 1.10 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	3.85 ms 0.002 ms/sec 0.064 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	501 0.25/sec 1.00/call 0.01%	221,702.28 ms 110.735 ms/sec 442.520 ms/call 8.22%	59,105 29.52/sec 117.97/call 0.16%	158,519,068 blks 79.18K blks/sec 316.41K blks/call 11.67%	140 blks 0.07 blks/sec 0.28 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	7.80 ms 0.004 ms/sec 0.016 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	509 0.25/sec 1.00/call 0.01%	219,096.76 ms 109.434 ms/sec 430.446 ms/call 8.12%	61,072 30.50/sec 119.98/call 0.17%	161,644,102 blks 80.74K blks/sec 317.58K blks/call 11.90%	150 blks 0.07 blks/sec 0.29 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	7.84 ms 0.004 ms/sec 0.015 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id))) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	25 0.01/sec 1.00/call 0.00%	173,803.25 ms 86.811 ms/sec 6952.130 ms/call 6.44%	1 0.00/sec 0.04/call 0.00%	75,796,794 blks 37.86K blks/sec 3.04M blks/call 5.58%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	28 0.01/sec 1.00/call 0.00%	120,869.77 ms 60.372 ms/sec 4316.778 ms/call 4.48%	28 0.01/sec 1.00/call 0.00%	10,817,061 blks 5.41K blks/sec 386.33K blks/call 0.80%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	232 0.12/sec 1.00/call 0.00%	83,328.35 ms 41.621 ms/sec 359.174 ms/call 3.09%	10,010 5.00/sec 43.15/call 0.03%	62,013,016 blks 30.98K blks/sec 267.30K blks/call 4.56%	31 blks 0.02 blks/sec 0.13 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	22.79 ms 0.011 ms/sec 0.098 ms/call 0.02%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 3204454 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) AND "projects"."pending_delete" = FALSE ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT 21 OFFSET 0 Full query	10 0.00/sec 1.00/call 0.00%	76,929.08 ms 38.424 ms/sec 7692.908 ms/call 2.85%	210 0.10/sec 21.00/call 0.00%	3,863,477 blks 1.93K blks/sec 386.35K blks/call 0.28%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? ORDER BY "environments"."id" ASC LIMIT ? OFFSET ? Full query	485 0.24/sec 1.00/call 0.00%	53,252.40 ms 26.598 ms/sec 109.799 ms/call 1.97%	35,133 17.55/sec 72.44/call 0.10%	50,356,132 blks 25.16K blks/sec 103.83K blks/call 3.71%	6 blks 0.00 blks/sec 0.01 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.09 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = ? AND "deployments"."status" = ? ORDER BY deployments.id DESC LIMIT ? Full query	12,079 6.03/sec 1.00/call 0.12%	52,214.55 ms 26.080 ms/sec 4.323 ms/call 1.94%	9,789 4.89/sec 0.81/call 0.03%	17,763,784 blks 8.88K blks/sec 1.48K blks/call 1.31%	2,331 blks 1.16 blks/sec 0.19 blks/call 0.14%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	193.68 ms 0.097 ms/sec 0.016 ms/call 0.14%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	993 0.50/sec 1.00/call 0.01%	51,232.67 ms 25.590 ms/sec 51.594 ms/call 1.90%	993 0.50/sec 1.00/call 0.00%	3,780,351 blks 1.89K blks/sec 3.81K blks/call 0.28%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	7,144 3.57/sec 1.00/call 0.07%	50,994.65 ms 25.471 ms/sec 7.138 ms/call 1.89%	1,988 0.99/sec 0.28/call 0.01%	21,924,093 blks 10.96K blks/sec 3.07K blks/call 1.61%	19,321 blks 9.65 blks/sec 2.70 blks/call 1.14%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	950.34 ms 0.475 ms/sec 0.133 ms/call 0.70%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	790,970 395.07/sec 1.00/call 8.06%	48,426.92 ms 24.188 ms/sec 0.061 ms/call 1.80%	779,687 389.44/sec 0.99/call 2.12%	7,076,989 blks 3.54K blks/sec 8.95 blks/call 0.52%	9,626 blks 4.81 blks/sec 0.01 blks/call 0.57%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	510.97 ms 0.255 ms/sec 0.001 ms/call 0.38%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("issues"."title" ILIKE ? OR "issues"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	10 0.00/sec 1.00/call 0.00%	44,413.51 ms 22.184 ms/sec 4441.351 ms/call 1.65%	10 0.00/sec 1.00/call 0.00%	807,463 blks 403.31 blks/sec 80.75K blks/call 0.06%	53,736 blks 26.84 blks/sec 5.38K blks/call 3.16%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1,166.33 ms 0.583 ms/sec 116.633 ms/call 0.86%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT "users".* FROM "users" WHERE ("users"."state" IN (?)) AND (ghost IS NOT TRUE) AND "users"."bot_type" IS NULL AND (("users"."name" ILIKE ? OR "users"."username" ILIKE ?) OR "users"."email" = ?) ORDER BY CASE WHEN users.name = ? THEN ? WHEN users.username = ? THEN ? WHEN users.email = ? THEN ? ELSE ? END , "users"."name" ASC LIMIT ? Full query	485 0.24/sec 1.00/call 0.00%	37,636.98 ms 18.799 ms/sec 77.602 ms/call 1.40%	4,770 2.38/sec 9.84/call 0.01%	3,581,205 blks 1.79K blks/sec 7.39K blks/call 0.26%	2,334 blks 1.17 blks/sec 4.81 blks/call 0.14%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	125.88 ms 0.063 ms/sec 0.260 ms/call 0.09%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	82 0.04/sec 1.00/call 0.00%	34,724.29 ms 17.344 ms/sec 423.467 ms/call 1.29%	8,282 4.14/sec 101.00/call 0.02%	9,033,895 blks 4.52K blks/sec 110.17K blks/call 0.66%	1 blks 0.00 blks/sec 0.01 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.03 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() ; Full query	134 0.07/sec 1.00/call 0.00%	31,221.79 ms 15.595 ms/sec 232.998 ms/call 1.16%	134 0.07/sec 1.00/call 0.00%	19,671,348 blks 9.83K blks/sec 146.81K blks/call 1.45%	4 blks 0.00 blks/sec 0.03 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.75 ms 0.000 ms/sec 0.006 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() - ?::interval ; Full query	134 0.07/sec 1.00/call 0.00%	29,784.09 ms 14.877 ms/sec 222.269 ms/call 1.10%	134 0.07/sec 1.00/call 0.00%	19,530,240 blks 9.76K blks/sec 145.75K blks/call 1.44%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = ? ORDER BY "ci_pipelines"."id" DESC LIMIT ? OFFSET ? Full query	4,099 2.05/sec 1.00/call 0.04%	29,491.30 ms 14.730 ms/sec 7.195 ms/call 1.09%	78,574 39.25/sec 19.17/call 0.21%	4,583,312 blks 2.29K blks/sec 1.12K blks/call 0.34%	365,541 blks 182.58 blks/sec 89.18 blks/call 21.52%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	10,950.43 ms 5.470 ms/sec 2.671 ms/call 8.12%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT projects.id FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	5 0.00/sec 1.00/call 0.00%	25,572.89 ms 12.773 ms/sec 5114.578 ms/call 0.95%	0 0.00/sec 0.00/call 0.00%	1,545,484 blks 771.94 blks/sec 309.10K blks/call 0.11%	10 blks 0.00 blks/sec 2.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.11 ms 0.000 ms/sec 0.022 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT COUNT(*) FROM (SELECT ? AS one FROM "merge_requests" INNER JOIN "projects" ON "projects"."id" = "merge_requests"."target_project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."merge_requests_access_level" IS NULL OR "project_features"."merge_requests_access_level" IN (?,?) OR ("project_features"."merge_requests_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("merge_requests"."title" ILIKE ? OR "merge_requests"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	10 0.00/sec 1.00/call 0.00%	25,397.40 ms 12.685 ms/sec 2539.740 ms/call 0.94%	10 0.00/sec 1.00/call 0.00%	521,474 blks 260.46 blks/sec 52.15K blks/call 0.04%	159,951 blks 79.89 blks/sec 16.00K blks/call 9.41%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2,602.06 ms 1.300 ms/sec 260.206 ms/call 1.93%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.107`)

Start: 2019-11-11T13:05:38.708195+00:00
End: 2019-11-11T13:43:26.762667+00:00
Period seconds: 2268.05448
Period age: 00:37:48.054472

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = 'Namespace' AND "namespaces"."type" IN ('Group') AND "members"."user_id" = 92192 AND "members"."requested_at" IS NULL AND (members.access_level >= 30)) UNION (SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND (EXISTS (SELECT 1 FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN ('silver... Full query	58 0.03/sec 1.00/call 0.00%	519,696.03 ms 229.137 ms/sec 8960.276 ms/call 14.24%	58 0.03/sec 1.00/call 0.00%	500,111,851 blks 220.51K blks/sec 8.63M blks/call 28.50%	79 blks 0.03 blks/sec 1.36 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2.16 ms 0.001 ms/sec 0.037 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND ("ci_builds"."status" IN ('running')) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = 1) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND ("ci_builds"."status" IN ('pending')) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = TRUE AND "projects"."pending_delete" = FALSE AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) AND (projects.visibility_level=20 OR (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces". FROM "namespaces" WHE... Full query	622 0.27/sec 1.00/call 0.01%	253,390.23 ms 111.721 ms/sec 407.380 ms/call 6.94%	68,628 30.26/sec 110.33/call 0.16%	198,292,887 blks 87.43K blks/sec 318.80K blks/call 11.30%	167 blks 0.07 blks/sec 0.27 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	4.96 ms 0.002 ms/sec 0.008 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id)) ... Full query	556 0.25/sec 1.00/call 0.00%	226,697.36 ms 99.952 ms/sec 407.729 ms/call 6.21%	57,859 25.51/sec 104.06/call 0.13%	176,977,507 blks 78.04K blks/sec 318.31K blks/call 10.09%	153 blks 0.07 blks/sec 0.28 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	25.51 ms 0.011 ms/sec 0.046 ms/call 0.03%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id)))) UNION (SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	32 0.01/sec 1.00/call 0.00%	205,312.82 ms 90.524 ms/sec 6416.026 ms/call 5.62%	0 0.00/sec 0.00/call 0.00%	100,064,679 blks 44.12K blks/sec 3.13M blks/call 5.70%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	43 0.02/sec 1.00/call 0.00%	149,399.57 ms 65.871 ms/sec 3474.409 ms/call 4.09%	43 0.02/sec 1.00/call 0.00%	16,611,871 blks 7.33K blks/sec 386.33K blks/call 0.95%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 644481 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) AND "projects"."pending_delete" = FALSE ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT 21 OFFSET 0 Full query	18 0.01/sec 1.00/call 0.00%	125,014.86 ms 55.120 ms/sec 6945.270 ms/call 3.43%	378 0.17/sec 21.00/call 0.00%	6,954,156 blks 3.07K blks/sec 386.35K blks/call 0.40%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT COUNT(*) FROM (SELECT ? AS one FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."pending_delete" = ? LIMIT ?) subquery_for_count Full query	9 0.00/sec 1.00/call 0.00%	114,629.95 ms 50.541 ms/sec 12736.661 ms/call 3.14%	9 0.00/sec 1.00/call 0.00%	51,543,837 blks 22.73K blks/sec 5.73M blks/call 2.94%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	93 0.04/sec 1.00/call 0.00%	100,694.91 ms 44.397 ms/sec 1082.741 ms/call 2.76%	9,340 4.12/sec 100.43/call 0.02%	41,173,772 blks 18.16K blks/sec 442.73K blks/call 2.35%	3,114 blks 1.37 blks/sec 33.48 blks/call 0.16%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	49.14 ms 0.022 ms/sec 0.528 ms/call 0.05%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT COUNT(*) FROM (SELECT ? AS one FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? LIMIT ?) subquery_for_count Full query	51 0.02/sec 1.00/call 0.00%	89,615.93 ms 39.512 ms/sec 1757.175 ms/call 2.46%	51 0.02/sec 1.00/call 0.00%	40,560,043 blks 17.89K blks/sec 795.30K blks/call 2.31%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("issues"."title" ILIKE ? OR "issues"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	20 0.01/sec 1.00/call 0.00%	89,070.90 ms 39.272 ms/sec 4453.545 ms/call 2.44%	20 0.01/sec 1.00/call 0.00%	1,813,389 blks 799.54 blks/sec 90.67K blks/call 0.10%	86,868 blks 38.30 blks/sec 4.35K blks/call 4.35%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1,792.97 ms 0.791 ms/sec 89.648 ms/call 1.81%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE ( issues.confidential IS NOT TRUE OR (issues.confidential = ? AND (issues.author_id = ? OR EXISTS (SELECT ? FROM issue_assignees WHERE user_id = ? AND issue_id = issues.id) OR EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = issues.project_id) AND (project_authorizations.access_level >= ?))))) AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_l... Full query	15 0.01/sec 1.00/call 0.00%	79,453.64 ms 35.032 ms/sec 5296.910 ms/call 2.18%	15 0.01/sec 1.00/call 0.00%	1,697,681 blks 748.52 blks/sec 113.18K blks/call 0.10%	13 blks 0.01 blks/sec 0.87 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.13 ms 0.000 ms/sec 0.008 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT "projects"."namespace_id" FROM "projects" WHERE (EXISTS ( SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	6 0.00/sec 1.00/call 0.00%	78,366.91 ms 34.552 ms/sec 13061.151 ms/call 2.15%	12 0.01/sec 2.00/call 0.00%	34,349,325 blks 15.15K blks/sec 5.73M blks/call 1.96%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id)) ... Full query	215 0.09/sec 1.00/call 0.00%	76,372.84 ms 33.673 ms/sec 355.223 ms/call 2.09%	9,718 4.28/sec 45.20/call 0.02%	57,596,039 blks 25.40K blks/sec 267.89K blks/call 3.28%	37 blks 0.02 blks/sec 0.17 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	15.17 ms 0.007 ms/sec 0.071 ms/call 0.02%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	6 0.00/sec 1.00/call 0.00%	75,771.06 ms 33.408 ms/sec 12628.509 ms/call 2.08%	600 0.26/sec 100.00/call 0.00%	34,348,076 blks 15.15K blks/sec 5.73M blks/call 1.96%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT "projects"."namespace_id" FROM "projects" WHERE (EXISTS ( SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	48 0.02/sec 1.00/call 0.00%	74,911.51 ms 33.029 ms/sec 1560.657 ms/call 2.05%	64 0.03/sec 1.33/call 0.00%	33,376,652 blks 14.72K blks/sec 695.35K blks/call 1.90%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT projects.id FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	48 0.02/sec 1.00/call 0.00%	74,531.32 ms 32.861 ms/sec 1552.736 ms/call 2.04%	10 0.00/sec 0.21/call 0.00%	33,375,711 blks 14.72K blks/sec 695.33K blks/call 1.90%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	8,295 3.66/sec 1.00/call 0.07%	59,859.90 ms 26.393 ms/sec 7.216 ms/call 1.64%	1,918 0.85/sec 0.23/call 0.00%	28,336,163 blks 12.50K blks/sec 3.42K blks/call 1.61%	13,014 blks 5.74 blks/sec 1.57 blks/call 0.65%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	386.98 ms 0.171 ms/sec 0.047 ms/call 0.39%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? ORDER BY "environments"."id" ASC LIMIT ? OFFSET ? Full query	604 0.27/sec 1.00/call 0.01%	56,981.99 ms 25.124 ms/sec 94.341 ms/call 1.56%	44,733 19.72/sec 74.06/call 0.10%	57,693,400 blks 25.44K blks/sec 95.52K blks/call 3.29%	5 blks 0.00 blks/sec 0.01 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.08 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	1,201 0.53/sec 1.00/call 0.01%	56,023.51 ms 24.701 ms/sec 46.647 ms/call 1.53%	1,201 0.53/sec 1.00/call 0.00%	4,572,207 blks 2.02K blks/sec 3.81K blks/call 0.26%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	893,930 394.14/sec 1.00/call 7.40%	53,266.93 ms 23.486 ms/sec 0.060 ms/call 1.46%	881,139 388.50/sec 0.99/call 2.05%	7,999,191 blks 3.53K blks/sec 8.95 blks/call 0.46%	10,559 blks 4.66 blks/sec 0.01 blks/call 0.53%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	343.79 ms 0.152 ms/sec 0.000 ms/call 0.35%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.108`)

Start: 2019-11-11T13:05:39.790748+00:00
End: 2019-11-11T13:45:48.057645+00:00
Period seconds: 2408.2669
Period age: 00:40:08.266897

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "members"."source_type" = ? AND "namespaces"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND (members.access_level >= ?) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces... Full query	64 0.03/sec 1.00/call 0.00%	592,134.63 ms 245.876 ms/sec 9252.104 ms/call 15.83%	64 0.03/sec 1.00/call 0.00%	510,518,962 blks 211.99K blks/sec 7.98M blks/call 30.13%	286 blks 0.12 blks/sec 4.47 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	3.93 ms 0.002 ms/sec 0.061 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	701 0.29/sec 1.00/call 0.01%	313,513.76 ms 130.182 ms/sec 447.238 ms/call 8.38%	64,863 26.93/sec 92.53/call 0.14%	224,201,503 blks 93.10K blks/sec 319.84K blks/call 13.23%	145 blks 0.06 blks/sec 0.21 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	6.74 ms 0.003 ms/sec 0.010 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	672 0.28/sec 1.00/call 0.01%	298,096.74 ms 123.781 ms/sec 443.596 ms/call 7.97%	67,034 27.83/sec 99.75/call 0.14%	213,782,176 blks 88.78K blks/sec 318.13K blks/call 12.62%	202 blks 0.08 blks/sec 0.30 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	8.00 ms 0.003 ms/sec 0.012 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id))) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	45 0.02/sec 1.00/call 0.00%	290,764.49 ms 120.736 ms/sec 6461.433 ms/call 7.77%	4 0.00/sec 0.09/call 0.00%	131,451,233 blks 54.59K blks/sec 2.93M blks/call 7.76%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	22 0.01/sec 1.00/call 0.00%	165,323.65 ms 68.648 ms/sec 7514.712 ms/call 4.42%	462 0.19/sec 21.00/call 0.00%	8,499,708 blks 3.53K blks/sec 386.36K blks/call 0.50%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	33 0.01/sec 1.00/call 0.00%	132,807.52 ms 55.147 ms/sec 4024.470 ms/call 3.55%	33 0.01/sec 1.00/call 0.00%	12,748,733 blks 5.30K blks/sec 386.33K blks/call 0.75%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	227 0.09/sec 1.00/call 0.00%	90,826.77 ms 37.715 ms/sec 400.118 ms/call 2.43%	10,185 4.23/sec 44.87/call 0.02%	61,103,146 blks 25.38K blks/sec 269.18K blks/call 3.61%	35 blks 0.01 blks/sec 0.15 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	16.68 ms 0.007 ms/sec 0.073 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? ORDER BY "environments"."id" ASC LIMIT ? OFFSET ? Full query	692 0.29/sec 1.00/call 0.01%	85,036.15 ms 35.310 ms/sec 122.885 ms/call 2.27%	53,275 22.12/sec 76.99/call 0.11%	82,160,862 blks 34.12K blks/sec 118.73K blks/call 4.85%	4 blks 0.00 blks/sec 0.01 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.09 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT COUNT(*) FROM (SELECT 1 AS one FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 3099500 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) AND "projects"."visibility_level" = 0 AND "projects"."pending_delete" = FALSE LIMIT 10001) subquery_for_count Full query	6 0.00/sec 1.00/call 0.00%	83,946.52 ms 34.858 ms/sec 13991.087 ms/call 2.24%	6 0.00/sec 1.00/call 0.00%	34,324,748 blks 14.26K blks/sec 5.73M blks/call 2.03%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT COUNT(*) FROM (SELECT ? AS one FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? LIMIT ?) subquery_for_count Full query	50 0.02/sec 1.00/call 0.00%	73,161.95 ms 30.380 ms/sec 1463.239 ms/call 1.96%	50 0.02/sec 1.00/call 0.00%	29,083,115 blks 12.08K blks/sec 581.67K blks/call 1.72%	1 blks 0.00 blks/sec 0.02 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.01 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	1,346 0.56/sec 1.00/call 0.01%	67,838.25 ms 28.169 ms/sec 50.400 ms/call 1.81%	1,346 0.56/sec 1.00/call 0.00%	5,124,222 blks 2.13K blks/sec 3.81K blks/call 0.30%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."user_id" = 1089588 AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT projects.id FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 1089588 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) AND "projects"."pending_delete" = FALSE ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT 5 OFFSET 30) Full query	11 0.00/sec 1.00/call 0.00%	66,515.43 ms 27.620 ms/sec 6046.858 ms/call 1.78%	0 0.00/sec 0.00/call 0.00%	4,250,000 blks 1.77K blks/sec 386.37K blks/call 0.25%	11 blks 0.00 blks/sec 1.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.17 ms 0.000 ms/sec 0.016 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 3011449 AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (10,20)) AND "projects"."visibility_level" = 0 AND "projects"."archived" = FALSE AND "projects"."pending_delete" = FALSE ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT 100 OFFSET 0 Full query	92 0.04/sec 1.00/call 0.00%	65,553.96 ms 27.220 ms/sec 712.543 ms/call 1.75%	9,266 3.85/sec 100.72/call 0.02%	21,363,315 blks 8.88K blks/sec 232.21K blks/call 1.26%	1 blks 0.00 blks/sec 0.01 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.05 ms 0.000 ms/sec 0.001 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	8,545 3.55/sec 1.00/call 0.07%	64,625.61 ms 26.835 ms/sec 7.563 ms/call 1.73%	1,976 0.82/sec 0.23/call 0.00%	26,857,554 blks 11.16K blks/sec 3.15K blks/call 1.59%	17,268 blks 7.17 blks/sec 2.02 blks/call 0.88%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	634.81 ms 0.264 ms/sec 0.074 ms/call 0.44%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND (ref = ?) GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	3,067 1.27/sec 1.00/call 0.02%	63,553.33 ms 26.390 ms/sec 20.722 ms/call 1.70%	1,290 0.54/sec 0.42/call 0.00%	30,709,354 blks 12.76K blks/sec 10.02K blks/call 1.81%	9,357 blks 3.89 blks/sec 3.05 blks/call 0.47%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	839.30 ms 0.349 ms/sec 0.274 ms/call 0.58%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	949,582 394.30/sec 1.00/call 7.30%	60,044.72 ms 24.933 ms/sec 0.063 ms/call 1.60%	935,892 388.62/sec 0.99/call 2.01%	8,496,840 blks 3.53K blks/sec 8.95 blks/call 0.50%	11,518 blks 4.78 blks/sec 0.01 blks/call 0.58%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	448.53 ms 0.186 ms/sec 0.000 ms/call 0.31%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT COUNT(*) FROM (SELECT ? AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = ? AND (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)) OR projects.visibility_level IN (?,?)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (?,?) OR ("project_features"."issues_access_level" = ? AND EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= ?)))) AND ("issues"."title" ILIKE ? OR "issues"."description" ILIKE ?) LIMIT ?) subquery_for_count Full query	15 0.01/sec 1.00/call 0.00%	58,602.24 ms 24.334 ms/sec 3906.816 ms/call 1.57%	15 0.01/sec 1.00/call 0.00%	1,066,936 blks 443.03 blks/sec 71.13K blks/call 0.06%	43,218 blks 17.95 blks/sec 2.89K blks/call 2.19%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	740.78 ms 0.308 ms/sec 49.386 ms/call 0.51%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT "projects"."namespace_id" FROM "projects" WHERE (EXISTS ( SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	11 0.00/sec 1.00/call 0.00%	54,863.05 ms 22.781 ms/sec 4987.550 ms/call 1.47%	0 0.00/sec 0.00/call 0.00%	3,477,368 blks 1.45K blks/sec 316.13K blks/call 0.21%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = ? AND "deployments"."status" = ? ORDER BY deployments.id DESC LIMIT ? Full query	13,710 5.69/sec 1.00/call 0.11%	48,241.83 ms 20.032 ms/sec 3.519 ms/call 1.29%	11,119 4.62/sec 0.81/call 0.02%	14,641,319 blks 6.08K blks/sec 1.07K blks/call 0.86%	1,740 blks 0.72 blks/sec 0.13 blks/call 0.09%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	112.76 ms 0.047 ms/sec 0.008 ms/call 0.08%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT "users".* FROM "users" WHERE ("users"."state" IN (?)) AND (ghost IS NOT TRUE) AND "users"."bot_type" IS NULL AND (("users"."name" ILIKE ? OR "users"."username" ILIKE ?) OR "users"."email" = ?) ORDER BY CASE WHEN users.name = ? THEN ? WHEN users.username = ? THEN ? WHEN users.email = ? THEN ? ELSE ? END , "users"."name" ASC LIMIT ? Full query	580 0.24/sec 1.00/call 0.00%	39,748.69 ms 16.505 ms/sec 68.532 ms/call 1.06%	5,305 2.20/sec 9.15/call 0.01%	3,777,261 blks 1.57K blks/sec 6.52K blks/call 0.22%	2,679 blks 1.11 blks/sec 4.62 blks/call 0.14%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	38.11 ms 0.016 ms/sec 0.066 ms/call 0.03%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Replica (`10.220.16.109`)

Start: 2019-11-11T13:05:40.877736+00:00
End: 2019-11-11T13:48:35.192983+00:00
Period seconds: 2574.31524
Period age: 00:42:54.315247

Top 50 rows

#	Query	Calls	▼ Total time	Rows	shared_blks_hit	shared_blks_read	shared_blks_dirtied	shared_blks_written	blk_read_time	blk_write_time	kcache_reads	kcache_writes	kcache_user_time_ms	kcache_system_time
1	SELECT COUNT() FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces". FROM "namespaces" INNER JOIN "members" ON "namespaces"."id" = "members"."source_id" WHERE "members"."type" IN (?) AND "members"."source_type" = ? AND "namespaces"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND (members.access_level >= ?) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT DISTINCT "namespaces"."id" FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces... Full query	72 0.03/sec 1.00/call 0.00%	813,278.29 ms 315.920 ms/sec 11295.532 ms/call 19.35%	72 0.03/sec 1.00/call 0.00%	642,494,291 blks 249.58K blks/sec 8.93M blks/call 35.65%	56 blks 0.02 blks/sec 0.78 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	1.02 ms 0.000 ms/sec 0.014 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
2	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	755 0.29/sec 1.00/call 0.01%	373,505.39 ms 145.089 ms/sec 494.709 ms/call 8.89%	64,729 25.14/sec 85.73/call 0.13%	241,763,098 blks 93.92K blks/sec 320.22K blks/call 13.41%	146 blks 0.06 blks/sec 0.19 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	9.25 ms 0.004 ms/sec 0.012 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
3	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	685 0.27/sec 1.00/call 0.01%	339,272.75 ms 131.791 ms/sec 495.289 ms/call 8.07%	63,287 24.58/sec 92.39/call 0.13%	218,368,286 blks 84.83K blks/sec 318.79K blks/call 12.12%	194 blks 0.08 blks/sec 0.28 blks/call 0.01%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	9.74 ms 0.004 ms/sec 0.014 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
4	WITH RECURSIVE "base_and_descendants" AS (SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN (?) AND (EXISTS (SELECT ? FROM "plans" INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id" WHERE "plans"."name" IN (?, ?) AND (gitlab_subscriptions.namespace_id = namespaces.id))) UNION SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN (?) AND "namespaces"."parent_id" = "base_and_descendants"."id") SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."id" = ? LIMIT ? Full query	41 0.02/sec 1.00/call 0.00%	337,354.69 ms 131.046 ms/sec 8228.163 ms/call 8.03%	1 0.00/sec 0.02/call 0.00%	126,718,469 blks 49.23K blks/sec 3.10M blks/call 7.03%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
5	SELECT COUNT(*) FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) Full query	48 0.02/sec 1.00/call 0.00%	208,665.74 ms 81.057 ms/sec 4347.203 ms/call 4.97%	48 0.02/sec 1.00/call 0.00%	18,543,577 blks 7.21K blks/sec 386.33K blks/call 1.03%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
6	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	19 0.01/sec 1.00/call 0.00%	150,566.71 ms 58.488 ms/sec 7924.564 ms/call 3.58%	399 0.15/sec 21.00/call 0.00%	7,341,062 blks 2.86K blks/sec 386.38K blks/call 0.41%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
7	SELECT COUNT(*) FROM (SELECT 1 AS one FROM "issues" INNER JOIN "projects" ON "projects"."id" = "issues"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id WHERE "issues"."confidential" = FALSE AND (EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 260391 AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= 10)) OR projects.visibility_level IN (10,20)) AND ("project_features"."issues_access_level" IS NULL OR "project_features"."issues_access_level" IN (20,30) OR ("project_features"."issues_access_level" = 10 AND EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = 260391 AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= 10)))) AND ("issues"."title" ILIKE '%Elektrifi%' OR "issues"."description" ILIKE '%Elektrifi%') LIMIT 101) subquery_for_count Full query	20 0.01/sec 1.00/call 0.00%	124,608.52 ms 48.405 ms/sec 6230.426 ms/call 2.97%	20 0.01/sec 1.00/call 0.00%	2,066,187 blks 802.62 blks/sec 103.31K blks/call 0.11%	108,342 blks 42.09 blks/sec 5.42K blks/call 4.63%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2,244.55 ms 0.872 ms/sec 112.228 ms/call 1.56%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
8	SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."project_id" LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."project_id", count() AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."runner_type" = ?) GROUP BY "ci_builds"."project_id") AS project_builds ON ci_builds.project_id=project_builds.project_id WHERE "ci_builds"."type" IN (?) AND ("ci_builds"."status" IN (?)) AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = ? AND "projects"."pending_delete" = ? AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > ?) AND (projects.visibility_level=? OR (WITH RECURSIVE "base_and_ancestors" AS (SELECT "namespaces". FROM "namespaces" WHERE (namespaces.id = projects.namespace_id) UN... Full query	244 0.09/sec 1.00/call 0.00%	104,706.03 ms 40.673 ms/sec 429.123 ms/call 2.49%	10,101 3.92/sec 41.40/call 0.02%	65,344,444 blks 25.39K blks/sec 267.81K blks/call 3.63%	35 blks 0.01 blks/sec 0.14 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	14.09 ms 0.005 ms/sec 0.058 ms/call 0.01%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
9	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? ORDER BY "environments"."id" ASC LIMIT ? OFFSET ? Full query	768 0.30/sec 1.00/call 0.01%	88,092.53 ms 34.220 ms/sec 114.704 ms/call 2.10%	60,417 23.47/sec 78.67/call 0.12%	77,212,300 blks 30.00K blks/sec 100.54K blks/call 4.28%	5 blks 0.00 blks/sec 0.01 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.07 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
10	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND (ref = ?) GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	3,280 1.27/sec 1.00/call 0.03%	86,321.98 ms 33.532 ms/sec 26.318 ms/call 2.05%	1,331 0.52/sec 0.41/call 0.00%	37,616,384 blks 14.62K blks/sec 11.47K blks/call 2.09%	5,284 blks 2.05 blks/sec 1.61 blks/call 0.23%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	269.42 ms 0.105 ms/sec 0.082 ms/call 0.19%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
11	SELECT COUNT(*) FROM "deploy_keys_projects" WHERE "deploy_keys_projects"."deploy_key_id" = ? Full query	1,481 0.58/sec 1.00/call 0.01%	77,228.51 ms 30.000 ms/sec 52.146 ms/call 1.84%	1,481 0.58/sec 1.00/call 0.00%	5,638,167 blks 2.20K blks/sec 3.81K blks/call 0.31%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
12	SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = ? AND ("environments"."state" IN (?)) AND "environments"."id" IN (SELECT "deployments"."environment_id" FROM "deployments" WHERE "deployments"."project_id" = ? AND "deployments"."sha" = ? GROUP BY "deployments"."environment_id") ORDER BY (SELECT MAX("deployments"."id") FROM "deployments" WHERE "deployments"."environment_id" = "environments"."id") ASC NULLS FIRST Full query	9,262 3.60/sec 1.00/call 0.07%	74,529.50 ms 28.951 ms/sec 8.047 ms/call 1.77%	2,032 0.79/sec 0.22/call 0.00%	29,027,436 blks 11.28K blks/sec 3.14K blks/call 1.61%	22,776 blks 8.85 blks/sec 2.46 blks/call 0.97%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	786.82 ms 0.306 ms/sec 0.085 ms/call 0.55%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
13	SELECT "projects"."id" AS t0_r0, "projects"."name" AS t0_r1, "projects"."path" AS t0_r2, "projects"."description" AS t0_r3, "projects"."created_at" AS t0_r4, "projects"."updated_at" AS t0_r5, "projects"."creator_id" AS t0_r6, "projects"."namespace_id" AS t0_r7, "projects"."last_activity_at" AS t0_r8, "projects"."import_url" AS t0_r9, "projects"."visibility_level" AS t0_r10, "projects"."archived" AS t0_r11, "projects"."merge_requests_template" AS t0_r12, "projects"."star_count" AS t0_r13, "projects"."merge_requests_rebase_enabled" AS t0_r14, "projects"."import_type" AS t0_r15, "projects"."import_source" AS t0_r16, "projects"."avatar" AS t0_r17, "projects"."approvals_before_merge" AS t0_r18, "projects"."reset_approvals_on_push" AS t0_r19, "projects"."merge_requests_ff_only_enabled" AS t0_r20, "projects"."issues_template" AS t0_r21, "projects"."mirror" AS t0_r22, "projects"."mirror_user_id" AS t0_r23, "projects"."shared_runners_enabled" AS t0_r24, "projects"."runners_token" AS t0_r25, "... Full query	1,016,766 394.97/sec 1.00/call 7.96%	67,837.07 ms 26.352 ms/sec 0.067 ms/call 1.61%	1,002,260 389.33/sec 0.99/call 2.04%	9,097,898 blks 3.54K blks/sec 8.95 blks/call 0.50%	12,413 blks 4.82 blks/sec 0.01 blks/call 0.53%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	506.30 ms 0.197 ms/sec 0.000 ms/call 0.35%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
14	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT projects.id FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	10 0.00/sec 1.00/call 0.00%	64,558.77 ms 25.078 ms/sec 6455.877 ms/call 1.54%	0 0.00/sec 0.00/call 0.00%	3,863,930 blks 1.51K blks/sec 386.40K blks/call 0.21%	1 blks 0.00 blks/sec 0.10 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.02 ms 0.000 ms/sec 0.002 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
15	SELECT "deployments".* FROM "deployments" WHERE "deployments"."environment_id" = ? AND "deployments"."status" = ? ORDER BY deployments.id DESC LIMIT ? Full query	16,910 6.57/sec 1.00/call 0.13%	60,467.86 ms 23.489 ms/sec 3.576 ms/call 1.44%	13,549 5.26/sec 0.80/call 0.03%	17,073,192 blks 6.64K blks/sec 1.01K blks/call 0.95%	2,454 blks 0.95 blks/sec 0.15 blks/call 0.10%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	161.39 ms 0.063 ms/sec 0.010 ms/call 0.11%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
16	SELECT "projects".* FROM "projects" WHERE (EXISTS (SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."visibility_level" = ? AND "projects"."archived" = ? AND "projects"."pending_delete" = ? ORDER BY "projects"."last_activity_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ? Full query	142 0.06/sec 1.00/call 0.00%	59,587.04 ms 23.147 ms/sec 419.627 ms/call 1.42%	14,342 5.57/sec 101.00/call 0.03%	15,647,218 blks 6.08K blks/sec 110.20K blks/call 0.87%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
17	SELECT "members".* FROM "members" WHERE "members"."source_type" = ? AND "members"."type" IN (?) AND "members"."user_id" = ? AND "members"."requested_at" IS NULL AND "members"."source_id" IN (SELECT "projects"."namespace_id" FROM "projects" WHERE (EXISTS ( SELECT ? FROM "project_authorizations" WHERE "project_authorizations"."user_id" = ? AND (project_authorizations.project_id = projects.id)) OR projects.visibility_level IN (?,?)) AND "projects"."pending_delete" = ? ORDER BY "projects"."created_at" DESC, "projects"."id" DESC LIMIT ? OFFSET ?) Full query	10 0.00/sec 1.00/call 0.00%	59,570.66 ms 23.140 ms/sec 5957.066 ms/call 1.42%	0 0.00/sec 0.00/call 0.00%	3,477,610 blks 1.36K blks/sec 347.77K blks/call 0.19%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
18	SELECT "users".* FROM "users" WHERE ("users"."state" IN (?)) AND (ghost IS NOT TRUE) AND "users"."bot_type" IS NULL AND (("users"."name" ILIKE ? OR "users"."username" ILIKE ?) OR "users"."email" = ?) ORDER BY CASE WHEN users.name = ? THEN ? WHEN users.username = ? THEN ? WHEN users.email = ? THEN ? ELSE ? END , "users"."name" ASC LIMIT ? Full query	618 0.24/sec 1.00/call 0.00%	50,017.91 ms 19.430 ms/sec 80.935 ms/call 1.19%	5,878 2.28/sec 9.51/call 0.01%	4,622,977 blks 1.80K blks/sec 7.49K blks/call 0.26%	2,862 blks 1.11 blks/sec 4.63 blks/call 0.12%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	66.66 ms 0.026 ms/sec 0.108 ms/call 0.05%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
19	SELECT COUNT(*) FROM projects INNER JOIN project_mirror_data ON project_mirror_data.project_id = projects.id INNER JOIN namespaces ON projects.namespace_id = namespaces.id LEFT JOIN plans ON namespaces.plan_id = plans.id WHERE projects.mirror = ? AND project_mirror_data.retry_count <= ? AND (projects.visibility_level = ? OR plans.name IN (?, ?, ?, ?)) AND project_mirror_data.status NOT IN (?, ?) AND project_mirror_data.next_execution_timestamp <= NOW() ; Full query	172 0.07/sec 1.00/call 0.00%	39,228.54 ms 15.238 ms/sec 228.073 ms/call 0.93%	172 0.07/sec 1.00/call 0.00%	25,295,141 blks 9.83K blks/sec 147.07K blks/call 1.40%	4 blks 0.00 blks/sec 0.02 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	2.06 ms 0.001 ms/sec 0.012 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%
20	SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = ? ORDER BY "ci_pipelines"."id" DESC LIMIT ? OFFSET ? Full query	5,186 2.01/sec 1.00/call 0.04%	38,660.16 ms 15.018 ms/sec 7.455 ms/call 0.92%	96,197 37.37/sec 18.55/call 0.20%	8,182,814 blks 3.18K blks/sec 1.58K blks/call 0.45%	501,401 blks 194.77 blks/sec 96.68 blks/call 21.44%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	0 blks 0.00 blks/sec 0.00 blks/call 0.00%	8,745.69 ms 3.397 ms/sec 1.686 ms/call 6.06%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 bytes 0.00 bytes/sec 0.00 bytes/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%	0.00 ms 0.000 ms/sec 0.000 ms/call 0.00%

Conclusions

[P1] For some query groups, total_time > 30.00% of overall timing, observed on: node: 10.220.16.106. Such a high percentage means that those queries are "major contributors" to resource consumption on those nodes. In other words, if a query group has total_time which is 30.00% of overall timing, it means that during the observation period, 30.00% of time CPUs were working on that node processing queries from this group.

Recommendations

[P1] For some query groups, total_time > 30.00% of overall timing. To reduce total_time for particular query group consider the following tactics:
- perform query micro-optimization (take particular query examples related to the group, use EXPLAIN and EXPLAIN (BUFFERS, ANALYZE) to optimize it; consider using Joe to boost the optimization process);
- if the frequency of execution is high (check the calls / second metric), try to find a way to reduce the frequency, changing the application code and/or, if applicable, applying caching.

Table of contents

L001 Table Sizes

Observations

Data collected: 2019-11-11 13:42:08 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

The list is limited to 20 items. Total: 339.

#	Table	Rows	▼ Total size	Table size	Index(es) Size	TOAST Size
	===== TOTAL =====	~9B	4.40 TiB	2.20 TiB	1.37 TiB	854.76 GiB
1	`merge_request_diff_files`	~995M	1.39 TiB (31.47%)	721.27 GiB (32.02%)	26.81 GiB (1.92%)	668.97 GiB (78.26%)
2	`ci_builds`	~327M	817.06 GiB (18.15%)	327.80 GiB (14.55%)	372.96 GiB (26.73%)	116.31 GiB (13.61%)
3	`merge_request_diff_commits`	~3B	681.08 GiB (15.13%)	523.79 GiB (23.26%)	143.55 GiB (10.29%)	13.75 GiB (1.61%)
4	`notes`	~208M	267.97 GiB (5.95%)	133.69 GiB (5.94%)	125.63 GiB (9.00%)	8.66 GiB (1.01%)
5	`ci_build_trace_sections`	~2B	222.06 GiB (4.93%)	108.64 GiB (4.82%)	113.43 GiB (8.13%)
6	`events`	~472M	122.85 GiB (2.73%)	31.32 GiB (1.39%)	91.54 GiB (6.56%)	8.00 KiB (0.00%)
7	`ci_job_artifacts`	~278M	93.92 GiB (2.09%)	41.50 GiB (1.84%)	52.43 GiB (3.76%)	8.00 KiB (0.00%)
8	`web_hook_logs`	~48M	92.62 GiB (2.06%)	56.21 GiB (2.50%)	3.92 GiB (0.28%)	32.50 GiB (3.80%)
9	`merge_requests`	~34M	71.05 GiB (1.58%)	14.78 GiB (0.66%)	55.05 GiB (3.94%)	1.23 GiB (0.14%)
10	`ci_pipelines`	~89M	67.08 GiB (1.49%)	18.34 GiB (0.81%)	48.70 GiB (3.49%)	33.58 MiB (0.00%)
11	`audit_events`	~202M	64.01 GiB (1.42%)	46.64 GiB (2.07%)	17.37 GiB (1.24%)	6.49 MiB (0.00%)
12	`issues`	~21M	60.13 GiB (1.34%)	13.40 GiB (0.59%)	41.04 GiB (2.94%)	5.69 GiB (0.66%)
13	`projects`	~12M	59.82 GiB (1.33%)	2.95 GiB (0.13%)	56.60 GiB (4.06%)	278.35 MiB (0.03%)
14	`sent_notifications`	~233M	52.67 GiB (1.17%)	33.35 GiB (1.48%)	19.33 GiB (1.38%)	8.00 KiB (0.00%)
15	`push_event_payloads`	~359M	50.50 GiB (1.12%)	42.83 GiB (1.90%)	7.67 GiB (0.55%)	8.00 KiB (0.00%)
16	`taggings`	~132M	35.25 GiB (0.78%)	8.45 GiB (0.38%)	26.80 GiB (1.92%)	8.00 KiB (0.00%)
17	`ci_stages`	~159M	34.76 GiB (0.77%)	13.21 GiB (0.59%)	21.55 GiB (1.54%)	8.00 KiB (0.00%)
18	`ci_builds_metadata`	~260M	32.35 GiB (0.72%)	12.83 GiB (0.57%)	19.53 GiB (1.40%)	8.00 KiB (0.00%)
19	`users`	~5M	26.32 GiB (0.58%)	2.06 GiB (0.09%)	24.27 GiB (1.74%)	176.00 KiB (0.00%)
20	`deployments`	~37M	20.73 GiB (0.46%)	5.79 GiB (0.26%)	14.95 GiB (1.07%)	8.00 KiB (0.00%)

Conclusions

Recommendations

Table of contents

L003 Integer (int2, int4) Out-of-range Risks in PKs

Observations

Data collected: 2019-11-11 13:42:10 +0000 UTC
Current database: gitlabhq_production

Master (`10.220.16.106`)

Table	PK	Type	Current max value	▼ Capacity used, %
`events`	`id`	int4	559,189,034	26.04
`ci_job_artifacts`	`id`	int4	359,066,904	16.72
`ci_builds`	`id`	int4	347,243,462	16.17
`ci_builds_metadata`	`id`	int4	271,219,059	12.63
`notes`	`id`	int4	242,855,645	11.31
`sent_notifications`	`id`	int4	232,647,086	10.83

Conclusions

[P1] High risks of out-of-range errors for an integer column. The columns listed below, being part of a primary key, have high risks to reach 100% of the integer capacity (2^31-1, or 2147483647 for int4 columns, and 2^15-1, or 32767 for int2 columns; see the documentation. If it happens, INSERTs of new rows will not be possible (unless they use some non-incremental values, such as some negative values) and fixing it will require a long downtime. 6 such columns are found:
- events: reached value 559189034, or 26.04% of int4 capacity
- ci_job_artifacts: reached value 359066904, or 16.72% of int4 capacity
- ci_builds: reached value 347243462, or 16.17% of int4 capacity
- ci_builds_metadata: reached value 271219059, or 12.63% of int4 capacity
- notes: reached value 242855645, or 11.31% of int4 capacity
- etc.

Recommendations

[P1] High risks of out-of-range errors for an integer column. Consider using int8 in all PK columns, always. To convert existing columns to int8, consider the following approaches:
1. Blocking ALTER TABLE .. ALTER COLUMN: a straightforward solution requiring significant downtime (a maintenance window).
2. "New column": create a new column, update it in batches (running not longer than a few seconds, to avoid blocking issues), and then switch to using it, redefining all the constraints. Notice, that to redefine a primary key constraint, ALTER TABLE .. ALTER COLUMN .. SET NOT NULL will be needed. It is a blocking operation in all Postgres versions up to 12 (where it might be lightweight if a proper CHECK constraint is defined first; such constraint can be defined in a non-blocking way). Since Postgres 11, it is possible to use a trick: when adding a column, use DEFAULT with NOT NULL, it will be a non-blocking operation. For all Postgres versions prior to 11, a specific downtime (maintenance window) will be needed anyway.
3. "New table": create a new table with the same schema as the existing one, capture all ongoing changes to an additional "log" table, copy existing data from the old table to the new one, and switch. This method, as the previous one, is non-trivial and requires careful development and testing under load (consider using Nancy for database experiments developing this solution). This approach is non-blocking regardless of Postgres version, but it requires significantly more efforts to implement.

postgres-checkup report for gitlab_production: 2019-11-11 (auto-generated)

PostgreSQL Checkup. Project: 'gitlab_production'. Database: 'gitlabhq_production'

Epoch number: '20191111001'

Table of contents

A002 Version Information

Observations

Master (10.220.16.106)

Replica servers:

Replica (10.220.16.101)

Replica (10.220.16.102)

Replica (10.220.16.103)

Replica (10.220.16.104)

Replica (10.220.16.105)

Replica (10.220.16.107)

Replica (10.220.16.108)

Replica (10.220.16.109)

Conclusions

Recommendations

A003 Postgres Settings

Observations

Master (10.220.16.106)

A004 Cluster Information

Observations

Databases sizes

Conclusions

Recommendations

A005 Extensions

Observations

Master (10.220.16.106)

Conclusions

Recommendations

A006 Postgres Setting Deviations

Observations

Settings (pg_settings) that Differ

Configs(pg_config) that differ

Conclusions

Recommendations

A007 Altered Settings

Observations

Master (10.220.16.106)

Replica servers:

Replica (10.220.16.101)

Replica (10.220.16.102)

Replica (10.220.16.103)

Replica (10.220.16.104)

Replica (10.220.16.105)

Replica (10.220.16.107)

Replica (10.220.16.108)

Replica (10.220.16.109)

Conclusions

Recommendations

D004 pg_stat_statements and pg_stat_kcache Settings

Observations

Master (10.220.16.106)

pg_stat_statements extension settings

Conclusions

Recommendations

F001 Autovacuum: Current Settings

Observations

Master (10.220.16.106)

Tuned tables

Replicas settings

Conclusions

Recommendations

F002 Autovacuum: Transaction ID Wraparound Check

Observations

Master (10.220.16.106)

Databases

Tables in the observed database

Conclusions

Recommendations

F003 Autovacuum: Dead Tuples

Observations

Master (10.220.16.106)

Conclusions

Recommendations

F004 Autovacuum: Heap Bloat (Estimated)

Observations

Master (10.220.16.106)

Conclusions

Master (`10.220.16.106`)

Replica (`10.220.16.101`)

Replica (`10.220.16.102`)

Replica (`10.220.16.103`)

Replica (`10.220.16.104`)

Replica (`10.220.16.105`)

Replica (`10.220.16.107`)

Replica (`10.220.16.108`)

Replica (`10.220.16.109`)

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Replica (`10.220.16.101`)

Replica (`10.220.16.102`)

Replica (`10.220.16.103`)

Replica (`10.220.16.104`)

Replica (`10.220.16.105`)

Replica (`10.220.16.107`)

Replica (`10.220.16.108`)

Replica (`10.220.16.109`)

Master (`10.220.16.106`)

`pg_stat_statements` extension settings

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Master (`10.220.16.106`)

Replica (`10.220.16.101`)

Replica (`10.220.16.102`)

Replica (`10.220.16.103`)

Replica (`10.220.16.104`)

Replica (`10.220.16.105`)

Replica (`10.220.16.107`)

Replica (`10.220.16.108`)

Replica (`10.220.16.109`)

Master (`10.220.16.106`)

Replica (`10.220.16.101`)

Replica (`10.220.16.102`)

Replica (`10.220.16.103`)

Replica (`10.220.16.104`)

Replica (`10.220.16.105`)

Replica (`10.220.16.107`)

Replica (`10.220.16.108`)

Replica (`10.220.16.109`)

Master (`10.220.16.106`)

Replica (`10.220.16.101`)

Replica (`10.220.16.102`)

Replica (`10.220.16.103`)

Replica (`10.220.16.104`)

Replica (`10.220.16.105`)

Replica (`10.220.16.107`)