Add ignore duplicates option to broken links test
Extracted from #84.
Add ignoreDuplicates
option to broken links test (defaults to true). If ignore duplicates is true and URL has been processed during this test run, return previous result.
Edited by Aaron Goldenthal