-N / --timestamping does not work perfectly
I was just trying to use Wget2 with --N enabled and realised that it downloads the file even if the server hasn't been modified. Wget works correctly in this case.
Wget2:
wget2 -N -d http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
15.105230.673 Local URI encoding = 'UTF-8'
15.105230.673 Input URI encoding = 'UTF-8'
15.105230.673 Fetched HSTS data from '/home/thedoctor/.wget-hsts'
15.105230.673 Fetched TLS session data from '/home/thedoctor/.wget-session'
15.105230.673 Fetched OCSP hosts from '/home/thedoctor/.wget-ocsp_hosts'
15.105230.673 Fetched OCSP fingerprints from '/home/thedoctor/.wget-ocsp'
15.105230.673 *url = http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
15.105230.673 *3 http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
15.105230.673 local filename = 'testing.db'
15.105230.673 host_add_job: job fname testing.db
15.105230.673 host_add_job: 0x2230c90 http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
15.105230.673 host_add_job: qsize 1 host-qsize=1
15.105230.673 queue_size: qsize=1
15.105230.673 queue_size: qsize=1
15.105230.673 queue_size: qsize=1
15.105230.673 [0] action=1 pending=0 host=0x0
15.105230.673 qsize=1 blocked=0
15.105230.673 pause=-1479203550673
15.105230.673 dequeue job http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
15.105230.673 resolving mirror.de.leaseweb.net:80...
15.105230.674 has 37.58.58.140:80
15.105230.674 has 2a00:c98:2030:a034::21:80
15.105230.674 Add dns cache entry mirror.de.leaseweb.net:80
15.105230.674 trying 37.58.58.140:80...
15.105230.674 established connection mirror.de.leaseweb.net
[0] Downloading 'http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db' ...
15.105230.675 cookie_create_request_header for host=mirror.de.leaseweb.net path=archlinux/testing/os/x86_64/testing.db
15.105230.689 # sent 307 bytes:
GET /archlinux/testing/os/x86_64/testing.db HTTP/1.1
Host: mirror.de.leaseweb.net
If-Modified-Since: Tue, 15 Nov 2016 08:46:48 GMT
Accept-Encoding: gzip, deflate, bzip2, xz, lzma
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: wget2/1.0.0
Connection: keep-alive
15.105230.689 [0] action=2 pending=1 host=0x2230b20
15.105230.689 ### req 0x7f1688005900 pending requests = 1
15.105230.705 nbytes 2760 nread 0 102400
15.105230.705 # got header 252 bytes:
HTTP/1.1 200 OK
Server: nginx/1.10.1
Date: Tue, 15 Nov 2016 09:52:30 GMT
Content-Type: application/octet-stream
Content-Length: 36214
Last-Modified: Tue, 15 Nov 2016 08:46:47 GMT
Connection: keep-alive
ETag: "582acb77-8d76"
Accept-Ranges: bytes
Saving 'testing.db'
15.105230.705 method 2
15.105230.707 nbytes 1380 total 3884/36214
15.105230.708 nbytes 1380 total 5264/36214
15.105230.710 nbytes 2760 total 8024/36214
15.105230.712 nbytes 2760 total 10784/36214
15.105230.712 nbytes 2760 total 13544/36214
15.105230.731 nbytes 2760 total 16304/36214
15.105230.732 nbytes 1380 total 17684/36214
15.105230.733 nbytes 2760 total 20444/36214
15.105230.734 nbytes 1380 total 21824/36214
15.105230.749 nbytes 1380 total 23204/36214
15.105230.750 nbytes 1380 total 24584/36214
15.105230.753 nbytes 1380 total 25964/36214
15.105230.756 nbytes 2760 total 28724/36214
15.105230.757 nbytes 1380 total 30104/36214
15.105230.757 nbytes 1380 total 31484/36214
15.105230.761 nbytes 4730 total 36214/36214
HTTP response 200 OK
15.105230.761 keep_alive=1
15.105230.761 host_remove_job: 0x2230c90
15.105230.761 host_remove_job: qsize=0 host->qsize=0
15.105230.761 [0] action=1 pending=0 host=0x2230b20
15.105230.761 qsize=0 blocked=0
15.105230.761 pause=-1479203550761
15.105230.761 closing connection
15.105230.761 [0] action=1 pending=0 host=0x0
15.105230.761 qsize=0 blocked=0
15.105230.761 pause=-1479203550761
15.105230.761 main: wake up
15.105230.761 main: done
15.105230.762 Successfully updated '/home/thedoctor/.wget-ocsp_hosts'.
15.105230.762 Saved OCSP hosts to '/home/thedoctor/.wget-ocsp_hosts'
15.105230.762 Successfully updated '/home/thedoctor/.wget-ocsp'.
15.105230.762 Saved OCSP fingerprints to '/home/thedoctor/.wget-ocsp'
15.105230.762 blacklist http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
Wget:
DEBUG output created by Wget 1.18.88-00ae-dirty on linux-gnu.
Reading HSTS entries from /home/thedoctor/.wget-hsts
Converted file name 'testing.db' (UTF-8) -> 'testing.db' (UTF-8)
--2016-11-15 10:53:07-- http://mirror.de.leaseweb.net/archlinux/testing/os/x86_64/testing.db
Resolving mirror.de.leaseweb.net... 37.58.58.140, 2a00:c98:2030:a034::21
Caching mirror.de.leaseweb.net => 37.58.58.140 2a00:c98:2030:a034::21
Connecting to mirror.de.leaseweb.net|37.58.58.140|:80... connected.
Created socket 4.
Releasing 0x0000000001d80e80 (new refcount 1).
---request begin---
GET /archlinux/testing/os/x86_64/testing.db HTTP/1.1
If-Modified-Since: Tue, 15 Nov 2016 08:46:47 GMT
User-Agent: Wget/1.18.88-00ae-dirty (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: mirror.de.leaseweb.net
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 304 Not Modified
Server: nginx/1.10.1
Date: Tue, 15 Nov 2016 09:53:07 GMT
Last-Modified: Tue, 15 Nov 2016 08:46:47 GMT
Connection: keep-alive
ETag: "582acb77-8d76"
---response end---
304 Not Modified
Registered socket 4 for persistent reuse.
File ‘testing.db’ not modified on server. Omitting download.
As you can see, for the same file, Wget2 is issuing a If-Modified-Since
header with timestamp of 08:46:48
while Wget issues it with a value of 08:46:47
. I'm not sure what is going wrong, but the server responds with a 200 OK to Wget2 and a 304 Not Modified to Wget which is the expected response.
I haven't looked into this much, could be a server issue, but am filing a bug here to remember to look into it at some point