Skip to content

[Geo] Docker image replication(2/4)

Valery Sizov requested to merge docker_image_replication into master

What does this MR do?

This MR adds Docker registry sync to a secondary node. The basic principle is that same as for files and git repositories. When we push a new image to the docker registry the notification event is created in the primary database and it will be propagated to the secondary. We intentionally don't track every pushed tag, instead, a secondary node makes a request to the primary's Registry API to get the list of available tags (with their digests) in this particular docker registry repository and with that information, the secondary pulls new images or removes outdated ones.

Why we don’t track updates on per image basis?

Because it creates some problems on the primary node, it will be needed to track all those tags and it will be also needed to actualize that information periodically because the docker registry notification system is not reliable at all. Also, that would require to maintain a huge database table and make SQL against it often so it’s unpractical.

Are there points in the code the reviewer needs to double check?

Why was this MR needed?

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

https://gitlab.com/gitlab-org/gitlab-ee/issues/2870

TODO:

  • Registry for containers
  • Setup events from registry
  • Implement even system for containers
  • Implement image sync
  • Implement sync for backfill condition (Dispatch worker)
  • Data model (tables, foreign keys)
  • GeoNodeStatus parameters calculation
  • Send MR to early review and fix comments
  • GeoNodeStatus parameters are shown in UI, rake tasks, API
  • GeoNodeStatus parameters are exposed in Prometheus
  • Notification has to be enabled when Geo is enabled
  • Turn off/on
  • Specs for everything
  • Add Lease for sync service
  • Admin documentation (separate MR)
  • Authentication in lib/api/container_registry_event.rb
Edited by Valery Sizov

Merge request reports