Improve Geo initial repository synchronization speed

Per https://gitlab.com/gitlab-org/gitlab-ce/issues/39345#note_57004192

In %10.6 we're introducing an endpoint that will provide a low-CPU way to get a tarball of a repository. This will be in a possibly inconsistent state, and will need a follow-up git fetch operation to get into a consistent state.

For ~Geo, we need to do two things:

  • Add JWT authentication to the endpoint we'll be adding as part of https://gitlab.com/gitlab-org/gitlab-ce/issues/39345
  • Rework https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/workers/geo/project_sync_worker.rb to use this endpoint for initial replication, instead of following the existing process.

The initial replication process should:

  • Download the .tar file, extracting it to the correct place on-the-fly
  • Perform a follow-up git fetch to make the repository consistent
  • (possibly) Run git fsck to verify success
Assignee Loading
Time tracking Loading