GNU Wget Couldn't Convert Links Properly

Dear wget developers!

The wget command failed to convert links into its local offline form after fetching a site recursively. Thus the -k option wasn't processed properly and the vast majority of links were left unchanged in their initial online form and the site after dowloading couldn't be serfed online.

The site downloaded was

https://www.eskimo.com/~scs/index.html

and the command I used to fetch it was

wget -e robots=off --no-check-certificate -t 20 -r -l 0 -np -p --follow-ftp -k 'https://www.eskimo.com/~scs/index.html'

and the version of wget I used was GNU Wget 1.21.1 built on msys. The Msys environment on its turn was launched on Windows 7.

The site was downloaded, but it doesn't work as an offline mirror. Could you kindly test this bug?

Best regards, Victor.

changed the description

Hi @victormmr,

Thanks for your report.

The site was downloaded, but it doesn't work as an offline mirror

To speed investigation up please add details (which page links were created not as expected) in the following way:

Expected: ...

Actual result: ...

Thanks.

Hello, Shamil!

It seems I was wrong when I claimed in my starting message that the vast majority of links in the mirror failed to be converted and stayed unchanged in their initial form. Most links were transformed the right way. Still some links in the mirror weren't converted properly and there's some inconvenience when using this mirror.

Here are some examples of these links.

File www.eskimo.com~scs\index.html

Expected: cclass/cclass.html
Actual result: https://www.eskimo.com/~scs/cclass/cclass.html

Expected: src/index.html
Actual result: https://www.eskimo.com/~scs/src/

Expected: dmr.html
Actual result: https://www.eskimo.com/~scs/dmr.html

Expected: readings/index2.html
Actual result: https://www.eskimo.com/~scs/readings/index2.html

Expected: links.html
Actual result: https://www.eskimo.com/~scs/links.html

and some more links on this page.

File www.eskimo.com~scs\dmr.html

Expected: index.html
Actual result: http://www.eskimo.com/~scs/

File www.eskimo.com/~scs/cclass/cclass.html

Expected: cclass/asgn.int/index.html
Actual result: http://www.eskimo.com/~scs/cclass/asgn.int/index.html

Expected: cclass/handouts/index.html
Actual result: http://www.eskimo.com/~scs/cclass/handouts

The "handouts" file (perhaps under the name of index.html) does not exist in the mirrored tree.

File www.eskimo.com~scs\cclass\int\top.html

Expected: index.html
Actual result: http://www.eskimo.com/~scs/

File www.eskimo.com~scs\cclass\int\sx10.html

Expected: index.html
Actual result: http://www.eskimo.com/~scs/

Hello, Viktor.

Thanks for clarifying, I do appreciate it! I will investigate this issue.

Hi,

Issue partially reproduced (macos), namely the absolute link to http scheme is stored as absolute one:

File www.eskimo.com\~scs\dmr.html Expected: index.html
Actual result: http://www.eskimo.com/~scs/ I'm still investigating this issue, will inform progress.

Also, the new issue found for the same command above with the ftp link following does not consider -H option.

Actual: Ftp scheme link from https scheme page is followed even if no -H/--span-hosts option is set. Expected: Ftp link following considers -H/--span-hosts option. Example (ftp://rtfm.mit.edu/* should not be fetched):

--2021-12-12 12:11:25--  https://www.eskimo.com/~scs/readings/portability.20031117.html
Reusing existing connection to www.eskimo.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 19321 (19K) [text/html]
Saving to: ‘www.eskimo.com/~scs/readings/portability.20031117.html’

www.eskimo.com/~scs/readings/portability. 100%[====================================================================================>]  18.87K  --.-KB/s    in 0.001s

2021-12-12 12:11:25 (16.7 MB/s) - ‘www.eskimo.com/~scs/readings/portability.20031117.html’ saved [19321/19321]

--2021-12-12 12:11:25--  ftp://rtfm.mit.edu/pub/usenet/news.answers/sci-data-formats
           => ‘rtfm.mit.edu/pub/usenet/news.answers/sci-data-formats’
Resolving rtfm.mit.edu (rtfm.mit.edu)... 18.25.130.75
Connecting to rtfm.mit.edu (rtfm.mit.edu)|18.25.130.75|:21... failed: Operation timed out.
Retrying.