One for the retrospective: this is clearly having an impact on customers as we're seeing various Marquee Customer Alerts firing, but the web service apdex has not been heavily affected at all:
I see a lot of inserts from Puma and Unicorn workers for SlackService. If it were a migration, then we would see this in Sidekiq. I suspect there's something lazily creating these services.
app/services/projects/create_service.rb:166:in`block in create_services_from_active_templates'app/services/projects/create_service.rb:164:in `create_services_from_active_templates'lib/gitlab/metrics/instrumentation.rb:161:in `block in create_services_from_active_templates'lib/gitlab/metrics/method_call.rb:36:in`measure'lib/gitlab/metrics/instrumentation.rb:161:in `create_services_from_active_templates'app/services/projects/create_service.rb:137:in `block in save_project_and_import_data'ee/lib/gitlab/database/load_balancing/connection_proxy.rb:79:in`block in write_using_load_balancer'ee/lib/gitlab/database/load_balancing/load_balancer.rb:84:in `blockinread_write'ee/lib/gitlab/database/load_balancing/load_balancer.rb:134:in `retry_with_backoff'ee/lib/gitlab/database/load_balancing/load_balancer.rb:83:in`read_write'ee/lib/gitlab/database/load_balancing/connection_proxy.rb:73:in `write_using_load_balancer'ee/lib/gitlab/database/load_balancing/connection_proxy.rb:47:in `block (2 levels) in <class:ConnectionProxy>'app/services/projects/create_service.rb:132:in`save_project_and_import_data'lib/gitlab/metrics/instrumentation.rb:161:in `blockinsave_project_and_import_data'lib/gitlab/metrics/method_call.rb:36:in `measure'lib/gitlab/metrics/instrumentation.rb:161:in`save_project_and_import_data'app/services/projects/create_service.rb:56:in `execute'ee/app/services/ee/projects/create_service.rb:17:in `execute'app/controllers/projects_controller.rb:58:in`create'
Is this just a mass import of projects, and the inserts arefrom creating services from templates?
Henri Philippschanged title from High insert rate for services table causing higher latencies to High insert rate for services table causing higher web latencies
changed title from High insert rate for services table causing higher latencies to High insert rate for services table causing higher web latencies
It looks like that we enabled creating instance templates for each project creation due to a bug, causing 12000 services to be created for each project creation (doing URL validation for each service, causing delays) which is causing the high db insert rate.
Henri Philippschanged title from High insert rate for services table causing higher web latencies to 2020-02-11: High insert rate for services table causing higher web latencies
changed title from High insert rate for services table causing higher web latencies to 2020-02-11: High insert rate for services table causing higher web latencies