Add an option to keep requests within the regexes / filter rules (wget1 backwards compatible)

... from the mailing list bug-wget@gnu.org (David Niklas)...

The arguments for the wget1's behavior are (in no particular order):

1: You don't want stuff that would be found by wget/wget2 if it was to search the files which are in the reject-regex.

2: The webmaster created a robots.txt that's a bit too strict. So you turn off robots but create a regex that's very similar. Thus, you don't want to get the stuff that in your reject-regex.

3: The webmaster failed to create a robots.txt file at all, but clearly the website would benefit from one. In this case, as you may notice from my reject-regex, I already have it setup to work in place of the more common rules in the robots files out there.

4: You're wasting your bandwidth and the servers on materials that you don't want.

Considering how many websites complain about robot user-agents or have notes about various annoying ones in their robots.txt files, I think the last argument is a very important one to make.