Propagate integrations using batching and queues

What does this MR do?

Follow-up from !40717 (merged), see conversations !40717 (comment 404570149) and !40717 (comment 404647371).

This is an important change in the architecture to propagate integrations. We can now propagate instance-level integrations and templates using batching and Sidekiq queues.

The problem before is the performance of the worst-case scenario, where there are no matching records and the anti-join. With the new approach, each job in the new queues handles a batch of projects/groups; rather than having a single job for all of them.

This is what we do right now in the complex case of propagating an instance-level integration:

• Update inherited integrations,
• Create integration for all projects without integration.
• Create integration for all groups without integration.

BEFORE:

                 Save integration
                        ↓
            ┌┬───────────────────────┬┐
            │| Propagate integration |│
            └┴───────────────────────┴┘
                        ↓
           Update inherited integrations
  Create integration for all projects without integration
  Create integration for all groups without integration

AFTER:

                                Save integration
                                        ↓
                            ┌┬───────────────────────┬┐
                            │| Propagate integration |│
                            └┴───────────────────────┴┘
                        ↓                ↓               ↓
  ┌┬─────────────────────┬┐ ┌┬───────────────────────┬┐ ┌┬───────────────────────────────┬┐
  │| Propagate to groups |│ │| Propagate to projects |│ │| Update inherited integrations |│
  └┴─────────────────────┴┘ └┴───────────────────────┴┘ └┴───────────────────────────────┴┘

Migration script

$ rails db:migrate
== 20200917165525 UpdateIndexOnNamespacesForTypeAndId: migrating ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:namespaces, [:type, :id], {:where=>"type IS NOT NULL", :name=>"index_namespaces_on_type_and_id_partial", :algorithm=>:concurrently})
   -> 0.0065s
-- add_index(:namespaces, [:type, :id], {:where=>"type IS NOT NULL", :name=>"index_namespaces_on_type_and_id_partial", :algorithm=>:concurrently})
   -> 0.0065s
-- transaction_open?()
   -> 0.0000s
-- indexes(:namespaces)
   -> 0.0049s
-- remove_index(:namespaces, {:algorithm=>:concurrently, :name=>"index_namespaces_on_type_partial"})
   -> 0.0054s
== 20200917165525 UpdateIndexOnNamespacesForTypeAndId: migrated (0.0241s) =====

$ rails db:migrate:down VERSION=20200917165525
== 20200917165525 UpdateIndexOnNamespacesForTypeAndId: reverting ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:namespaces, :type, {:where=>"type IS NOT NULL", :name=>"index_namespaces_on_type_partial", :algorithm=>:concurrently})
   -> 0.0075s
-- add_index(:namespaces, :type, {:where=>"type IS NOT NULL", :name=>"index_namespaces_on_type_partial", :algorithm=>:concurrently})
   -> 0.0038s
-- transaction_open?()
   -> 0.0000s
-- indexes(:namespaces)
   -> 0.0053s
-- remove_index(:namespaces, {:algorithm=>:concurrently, :name=>"index_namespaces_on_type_and_id_partial"})
   -> 0.0026s
== 20200917165525 UpdateIndexOnNamespacesForTypeAndId: reverted (0.0199s) =====

SQL queries

EXPLAIN SELECT MIN(namespaces.id),
       MAX(namespaces.id)
FROM "namespaces"
WHERE "namespaces"."type" = 'Group'
  AND (NOT EXISTS
         (SELECT 1
          FROM "services"
          WHERE (services.group_id = namespaces.id)
            AND "services"."type" = 'JiraService'))
  AND "namespaces"."id" >= 10000 AND "namespaces"."id" < 10100
LIMIT 1;

Time: 49.315 ms
  - planning: 0.663 ms
  - execution: 48.652 ms
    - I/O read: 47.713 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 37 (~296.00 KiB) from the buffer pool
  - reads: 26 (~208.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 19 (~152.00 KiB)
  - writes: 0

Query plan: https://explain.depesz.com/s/TwHb

---

SELECT "namespaces"."id"
FROM "namespaces"
WHERE "namespaces"."type" = 'Group'
  AND "namespaces"."id" BETWEEN 10000 AND 10100
  AND (NOT EXISTS
         (SELECT 1
          FROM "services"
          WHERE (services.group_id = namespaces.id)
            AND "services"."type" = 'JiraService'));

Time: 4.610 ms
  - planning: 0.555 ms
  - execution: 4.055 ms
    - I/O read: 3.867 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 27 (~216.00 KiB) from the buffer pool
  - reads: 8 (~64.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Query plan: https://explain.depesz.com/s/AsCX

---

SELECT MIN(projects.id),
       MAX(projects.id)
FROM "projects"
WHERE (NOT EXISTS
         (SELECT 1
          FROM "services"
          WHERE (services.project_id = projects.id)
            AND "services"."type" = 'JiraService'))
  AND "projects"."pending_delete" = FALSE
  AND "projects"."archived" = FALSE
  AND "projects"."id" >= 10000 AND "projects"."id" < 10100
LIMIT 1;

Time: 25.782 ms
  - planning: 1.091 ms
  - execution: 24.691 ms
    - I/O read: 24.198 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 219 (~1.70 MiB) from the buffer pool
  - reads: 14 (~112.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Query plan: https://explain.depesz.com/s/q2RE

---

SELECT "projects"."id"
FROM "projects"
WHERE "projects"."id" BETWEEN 10000 AND 10100
  AND (NOT EXISTS
         (SELECT 1
          FROM "services"
          WHERE (services.project_id = projects.id)
            AND "services"."type" = 'JiraService'))
  AND "projects"."pending_delete" = FALSE
  AND "projects"."archived" = FALSE;

Time: 1.112 ms
  - planning: 0.912 ms
  - execution: 0.200 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 236 (~1.80 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Query plan: https://explain.depesz.com/s/Xx4g

Related to #209831 (closed)
Closes #249564 (closed)