* tests/test.c: Add wget_robots_parse unit test
Created by: dstw
This is my GSoC 2017 microproject. To get more familiar with Wget2 codebase, with guide from Tim Ruehsen [1], I tried to create a simple unit test which call wget_robots_parse() from libwget to process data that contains robots.txt information.
[1] http://lists.gnu.org/archive/html/bug-wget/2017-03/msg00101.html