Add option to "enforce" compression

An option that makes wget2 stop downloading if the requested compression method(s) are ignored by the server.

Background HTTP doesn't know the "enforcing" of compression. The client just can inform the server what it supports. The server then can choose any of the offered compression types. The HTTP standard always implies "identity" (no compression) and has no way of saying "not identity". That leaves the user in a situation where it is impossible to enforce compression. Currently wget/wget2 offeer no way to avoid / error expensive uncompressed traffic.

Solution Add a new option --enforce-compression to avoid downloads if the server doesn't use one of the compression methods set via --compression=.

Downsides

  • won't really work with recursive downloads as short files are normally served uncompressed (compression may even make those files larger).
  • if the file is already compressed (e.g. a tar.gz), it may be sent uncompressed as additional compression doesn't make sense.
  • we have to think about Transport-Encoding: (badly supported by clients and servers)
  • there are lot's of buggy servers out there
  • should there be a content size threshold to not enforce compression for small files ?
  • is this just a corner case so that adding/maintaining extra code doesn't make sense (see https://github.com/curl/curl/issues/7516#) ? We have to weigh complexity against use cases.
Assignee Loading
Time tracking Loading