Integration settings not propagated to projects under a large group tree

Summary

Enabling integrations for a group with a large number of sub-groups / projects will not complete propagation to the projects.

Steps to reproduce

Note: I could only reproduce this in gitlab.com

1. Create a lot of sub-groups / projects.
   1. I used this script to generate 7 levels of subgroups with varying numbers of groups / projects. In total, there were 5000 projects and ~2200 subgroups.
2. In the parent group, enable an integration.
   1. I reproduced with both Microsoft Teams Notifications and Bugzilla.

Result:

Subgroups will show the integration and inherited settings from the parent group.

Projects will not show any inherited settings for the integration.

Example Project

https://gitlab.com/irisb_premium_group/655796

What is the current bug behavior?

The query to update integration settings for projects under the group times out

What is the expected correct behavior?

Projects would inherit the integration settings.

Relevant logs and/or screenshots

Sidekiq logs show failures for the class PropagateIntegrationWorker

PG::QueryCanceled: ERROR:  canceling statement due to statement timeout

	/*application:sidekiq,correlation_id:01K7JJ1A0XANQM8FHTW078K7FJ,jid:3cfeca082271439fc97d600b,endpoint_id:PropagateIntegrationWorker,db_config_database:gitlabhq_production_sidekiq,db_config_name:main*/ SELECT "projects"."id" FROM "projects" WHERE (NOT EXISTS (SELECT $1 FROM "integrations" WHERE (integrations.project_id = projects.id) AND "integrations"."type_new" = $2)) AND "projects"."pending_delete" = $3 AND "projects"."archived" = $4 AND "projects"."namespace_id" IN (SELECT "namespaces"."id" FROM "namespaces" WHERE "namespaces"."type" = $5 AND (traversal_ids @> ($6))) ORDER BY "projects"."id" ASC LIMIT $7

Postgres logs report that the query is cancelled due to timeout, which seems to be 15 seconds.

The logs from my reproduction are available here:

Correlation ID: 01K7JJ1A0XANQM8FHTW078K7FJ

Support Dashboard: https://log.gprd.gitlab.net/app/r/s/qZA5g

Postgres Logs: https://log.gprd.gitlab.net/app/r/s/wEHe3 

Stack trace

app/services/integrations/propagate_service.rb:84:in `propagate_integrations'
app/services/integrations/propagate_service.rb:77:in `create_integration_for_projects_without_integration_belonging_to_group'
app/services/integrations/propagate_service.rb:21:in `execute'
app/workers/propagate_integration_worker.rb:19:in `perform'

Output of checks

This bug happens on GitLab.com

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.