Improve download continuation (-c / --continue)

Wget allows to continue downloading with the -c / --continue option; it does so using the Range: request header.

Possible failure modes

  1. The server does not support the range header
  2. The file on the server changed it's content between the initial and the continuation download
  3. The file changes locally, e.g. due to user interaction, filesystem issues, etc
  4. The file contents change during the network transmission (bit flips, MITM, etc)

Proposed solutions

I would like to mention that there is a technology called Metalink, which deals with all of the failure modes and which gives a near-perfect user experience. Wget2 supports Metalink, though Metalink is not supported widely by servers.

So let's put Metalink aside and have a look how we can improve the situation without it.

For failure mode 1., there is nothing we can do but restart the download from the beginning. For failure mode 4., the solution is HTTP via TLS, (HTTPS, https://).

For the other failure modes (including 4. if HTTPS is not available), the user has to compare the file integrity, e.g. via checksum (md5, sha1, ...). The problem here is that this needs knowledge of the checksum (it must be provided, e.g. by the server / website). And even with this knowledge, the user can only generate and compare the checksum after the download has been completed. Some files may have self-contained checksums that allows for integrity checks by supporting applications (e.g. decompression may fail due to internal checksum errors) - but this knowledge is beyond the knowledge of wget.

Failure modes 2. and 3. can be treated the same: the local and the remote data do not match. The solutions here could be

  • the server's ETAG: header must match (where do we store the ETAG for a partially downloaded file, if not in extended attributes (which not all file systems support)). Not all servers support the ETAG: header.
  • the server's Last-Modified: header must match. (Same Q: where to store, not all servers provide it).
  • start the continuation some bytes earlier as needed to compare the overlapping bytes (must match). (How many bytes would be good in the general use case ?) If the initial data is below a certain number of bytes, we can forcefully restart the download from the beginning (What would be a threshold ?).