Consider using local temp storage instead of shared temp storage for Import/Export
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Discussion from: !32326 (comment 348497650)
does this directory need to be on shared storage, or could it just use TMPDIR? See https://docs.gitlab.com/ee/development/merge_request_performance_guidelines.html#local-temporary-storage.
That's an interesting question. Historically any temp files were on a shared storage for Import/Export. However, because Import/Export is performed on a singular sidekiq worker I would say there is no need for temp files to be on shared storage, since Import/Export is not distributed across multiple workers.
The only reason I would keep these files under shared storage is for a potential future change to have Import/Export process distributed across multiple workers. But because the current vision for Import/Export is to move away from files, I don't think we will do this change.
There might be another reason why these tmp files are under shared storage that I am not aware of.
Consider moving temp Import/Export files out of shared temp storage and use local temp storage instead, using Dir.mktmpdir. This will require wrapping the whole Import/Export process into a Dir.mktmpdir block, since tmp dir is removed once returned from the block.