Make final file hash faster through partial hash
As for now, Jirafeau makes a big md5 hash computation of the whole uploaded file at the very end of the download.
file_hash
option allows users to change this behavior by either partially md5 the file (which may be incorrect) or be totally random (fast but file deduplication does not work anymore).
An other option would be to hash each uploaded file chunks and incorporate the last hash so that:
chunk.0.md5 = md5(chunk.0.data)
chunk.1.md5 = md5(chunk.1.data + chunk.0.md5)
...
chunk.n.md5 = md5(chunk.n.data + chunk.n_1.md5)
pros:
- hash computation load is spread over each chunk uploads
- produce the same hash result for the same exact file
- upload finalization is very fast
cons:
- upload speed would probably drop
- partial failed uploads will consume CPU for nothing
Edited by Jérôme Jutteau