Multithreaded download: Change server probing behavior to better match browsers

Original issue at https://github.com/rockdaboot/wget2/issues/278

Chromium-based browsers, on media content (video embedded with the <video> tag), send a GET request with Range: bytes=0- and discard the body, instead of a HEAD request. I've seen servers that take advantage of it as a form of anti-scraping. Specifically, they disallow the use of HEAD requests, and return 404 for requests that don't have a Range header.

I propose that we add Range: bytes=0- to every initial GET request, and then check for 206 or the Accept-Ranges: bytes header to determine if segmented downloading is supported.

changed the description

I've seen servers that take advantage of it as a form of anti-scraping.

Can you provide an example ?

It sounds like a cat and mouse game. There are many different ways of doing anti-scraping. I assume the servers will change their method after most clients adopted a work-around for the 404 on HEAD requests and the Range requirement. I would also assume that some servers require a Range header to be not set and/or require a HEAD request before the GET.

So if we work-around this server behavior, I'd suggest to enable this via a new command line option.

Here's a server that has this behavior: https://videostream.askmath.vn/api/stream?id=f731ee20-4b9a-486b-9f9f-b85be4076b74&type=lesson&token=YU1+bnaji7tbREGs/60wIVDYxgZUpTxNQB3vYfwYiHxyWi2bL6KZdDlyurZeFMSaEMz6cwmBvrbBvXMocNZTTqF16jZMpaDl0nJn7Q2krEM=

(token embedded inside the link, if it expires let me know)

Required headers:

Accept: */*
Accept-Encoding: identity;q=1, *;q=0
Connection: keep-alive
Referer: https://vted.vn/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44

Here are some screenshots of testing that I've done

With HEAD requests: Server returns 405

Without Range headers: Server returns 404

With Range headers: Server returns 206 + data