Add ignore duplicates option to broken links test
Extracted from #84.
Add ignoreDuplicates option to broken links test (defaults to true). If ignore duplicates is true and URL has been processed during this test run, return previous result.
Edited by Aaron Goldenthal