review validation logic after put
At the moment dup_main calls a validate
after each put
to kind of check if the file was transferred correct.
Technically validate
means checking the file size of the file transferred to the backend and implemented in backend_wrapper only (even if backends could implement it.). But as there a several backends that doesn't even support the retrieval of the file size, this validate is kind of fail open and returns non blocking if for such backends.
On the other hand getting the correct file size need some retries until this information is reflects the current changes.
This design has some disadvantages.
- validation is optional and it is in-transparent to the user in which case the validation is done or not
- the validation logic is implemented at higher level, but need to catch corner cases from some backends e.g. if file size doesn't get refreshed instantly
- It is questionable if a backend has not a reliable
put
implementation, if it is the right implementation for a backup tool. - checking file size is nothing more than a light cross check, backends may have far better methods to check a successful copy
Some general thoughts about integrity checking in file transfer protocols:
- Threats
- unintentional bit flips because of hardware issues/cosmic rays etc.
- at read
- at transfer
- at write
- Interruption/Incomplete transfer
- unexpected errors at client, network, server
- implementation errors
- unintentional bit flips because of hardware issues/cosmic rays etc.
Most of the file transfer protocols are able to handle 1.2 and 2.1. Especially with encryption on the transport layer is a good counter measure for 1.2 .
But the rest is not really covered in most protocols. SFTP has an optional extension to request checksum, but it rarely implemented e.g. not implemented in OpenSSH. Servers like Next/Opencloud have some proprietary extensions for WebDav to get hashes. Checking filesize won't help for theses threats neither.
The most reliable way to check integrity of transferred files, is checking their GPG signature (assuming they were encrypted ). As a design pattern from dublicity is the independency of the backend/serverside, special checks can't be implemented there. The best fall back is using the verify
command, with the disadvantage of a whole download of all files.
Question: Is is worth to checking the file size, even if it is covering only a few threats, it is not possible for all backends and introduces code complexity(e.g. retries) , but may detect issues in duplicities logic? How get users aware of different quality of backends?
Proposal
- user should get a clear understanding of what duplicity is able to provide on this topic.
- may be this will differ per backend
- IMHO there should a a clear statement, that
verify
is the general option to check if the backup was successful.
- the main backup logic should work independent from the backend implementation.
- reliability at backend level
- either
- the put should do all task
- if size or whatever criteria is checked from duplicity, it must be accessible reliable (retry etc. must be implemented in the backend)
- either
Further considerations:
If concurrency is finally implemented :fingers_crossed:: do backup and validation in parallel instead of completing the whole backup first. Depending on the internet connection and server parallel up and download should work well side by side and lower the additional time required for the verification.
Fail ASAP: Any validation and verification should be done as soon as possible to fail fast. Many users transfer a huge amount of data each backup and duplicity can only recover from the first failed volume it would be helpful to stop the backup as soon as this can be detected. In other words: If the file size gets checked at the end of all transfers it could save time if all is successful but may double up the overall time, if a complete re-transfer is required.
This is just a starting point from the discussion on the mailing list. Comments, changes, implementations are welcome.