Improve our Geo finders

This is a follow up discussion from : #13158 (comment 205399162).

We have on the tracking database, our registry tables, which holds the state of each replicable items. One reason for why we some times have inconsistencies on admin UI when displaying the state of the replication is because we rely on sidekiq jobs to update the registry tables when we change/ enable anything on the partial replication parameters.

Here is an example:

  1. Start replication with 100 projects
  2. Wait for Geo to create the registries
  3. turn off sidekiq on the secondary
  4. Switch to partial replication

Other examples from previous discussion:

  1. We create a deletion event
  2. It is read by the log cursor, which triggers an async job
  3. The async job handles both the removal of the data and removal of the Registry
    • If the job fails, we retry upon to a limit, after that, we lost it forever.

Because the way we are querying stuff doesnt always use the same partial sync filters (for optimization reasons I believe), we will end up with a more than 100% synchronized items, until sidekiq process the changes.

We should think about a way to reduce or eliminate this kind of situation by always use the same filter base in every Finder.

This discussion is also related with the effort in #12565 (closed) as if we figure out a way to make the processing of events super reliable (in a way that we don't need to use the backfilling regularly), than we may be able to do something slightly smarter here (and perhaps use less brute force than a gigantic query).

Assignee Loading
Time tracking Loading