Skip to content

Use Geo log to remove files when migrated to object storage

Toon Claes requested to merge tc-geo-log-object-storage into master

What does this MR do?

When a file is migrated to object storage, it might already be synced to disk by the Geo secondary. These changes will generate a Geo Event log entry for these files, so the secondary can pick them up and remove them from local disk, if they did sync them.

CE backport: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22049

Manual tests

I'll run the rake tasks to migrate object manually to ensure each type generates an event:

Uploads

From the docs:

  • Avatars

    • Project avatar gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"

    • Group avatar gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"

    • User avatar gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"

  • Attachments

    • Note with markdown image gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"

    • Appearance logo gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"

    • Appearance header logo gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"

  • Favicon

    • Favicon gitlab-rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"
  • Markdown

    • File?? gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"

    • Personal snippet file gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"

    • Merge Request File gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"

CI Artifacts

From the docs.

  • CI Artifacts gitlab-rake gitlab:artifacts:migrate

LFS Objects

From the docs.

  • LFS Objects gitlab-rake gitlab:lfs:migrate

Are there points in the code the reviewer needs to double check?

Why was this MR needed?

Does this MR meet the acceptance criteria?

  • Changelog entry added, if necessary
  • Documentation created/updated
  • API support added
  • Tests added for this feature/bug
  • Conform by the code review guidelines
    • Has been reviewed by a UX Designer
    • Has been reviewed by a Frontend maintainer
    • Has been reviewed by a Backend maintainer
    • Has been reviewed by a Database specialist
  • EE specific content should be in the top level /ee folder
  • Conform by the merge request performance guides
  • Conform by the style guides
  • If you have multiple commits, please combine them into a few logically organized commits by squashing them
  • Internationalization required/considered
  • If paid feature, have we considered GitLab.com plan and how it works for groups and is there a design for promoting it to users who aren't on the correct plan
  • End-to-end tests pass (package-qa manual pipeline job)

What are the relevant issue numbers?

Closes gitlab-org/gitlab-ee#6040, gitlab-org/gitlab-ee#6135, and gitlab-org/gitlab-ee#7108

Continues on gitlab-org/gitlab-ee!4689

Edited by Toon Claes

Merge request reports