Skip to content

Geo: Expire and resync attachments from renamed projects in secondary nodes when using legacy storage

Gabriel Mazetto requested to merge 3674-attachments-legacy-storage-rename into master

What does this MR do?

This MR implements attachment expiration from secondary node, when project/namespace is moved/renamed.

We remove the file on the old location and mark the FileRegistry as failed, to trigger a resync in the new location.

Caveats

Depending on whether the secondary has replicated all the attachments or not, if this is triggered during any of these operations, we may not remove all the files as the list of files to remove may not be complete.

This is a fallback for when the user is not using the Hashed Storage yet. This method is not 100% consistent, to be 100% consistent we need a global lock, which we are not going to implement, as our final solution is the Hashed Storage.

Sidekiq cron Job that searches for files to download is re-scheduled at every 10 minutes, so in worst scenario, after expiring a file, it will take up to 10 minutes for it to start to be downloaded.

Are there points in the code the reviewer needs to double check?

We have distinct ways of retrieving data, whether is FDW or not. I tried my best to make both match, but please double check the queries :-)

Why was this MR needed?

We need to move files around on the secondary until the user migrates to the hashed storage.

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Part of #3674 (closed)

cc @stanhu @yorickpeterse

Edited by Gabriel Mazetto

Merge request reports