Sign in or sign up before continuing. Don't have an account yet? Register now to get started.
Register now

Geo: Implement consumer (aka GeoLogCursor) for event log

In #2384 (closed), we are adding a Geo event log and deprecating the use of system hooks. First, we should focus on these use cases:

  • git push
  • project deletion

We need to figure out how to consume these events properly and update the state of projects/files/directories.

We investigated LISTEN/NOTIFY channels in PostgreSQL to subscribe to changes in the event log, but this does not work well for Geo because:

  • The LISTEN and NOTIFY only works on clients connected to the primary, not the secondary (so we would have to have a process on the secondary Geo Node connect to the primary DB)
  • If the secondary is disconnected, it's possible the NOTIFY queue is filled (this is bad)

How this will work in the first iteration:

  • Poll the secondary database for changes in the latest ID
  • A separate process (Ruby to start, Go later if necessary) runs and scans this log
  • The consumer's job is to scan this log and update the repo/file state with CRUD information. For example, if we have a namespace with 100 projects deleted, we should mark all 100 projects as deleted and do a full sync of the paths affected by those actions.

Another process should then handle the actual resync based on the state of the tables.

@brodock @dbalexandre Does this make sense?

Edited May 23, 2017 by Stan Hu
Assignee Loading
Time tracking Loading