Wget2 plugin support implementation
This wiki documents my GSoc 2017 project entitled 'Design and implementation of framework for plugins'.
wget2 is a CLI HTTP client. This project added to wget2 ability to load plugins which then could influence its behavior.
Plugin support implementation was done in stages.
Stage 0: Plugin loading mechanism
This stage implemented loading plugins. Windows and platforms using libdl are supported. No useful plugin API was added in this stage.
--plugin command line option was added for loading an installed plugin.
User is also able to load plugins from arbitrary locations by
libwget/plugin.c: Plugin API.
src/plugin.c: Plugin management and API implementation.
src/dl.c: Abstraction for dynamic loading of object files.
wget2 plugins are shared libraries having
and linked agains libwget. wget2 calls
wget_plugin_initializer() on loading
a plugin and expects the function to register all the hooks the plugin is
Stage 1: Command line option forwarding
This stage implemented forwarding command line options to plugins.
Any option in form of
forwarded to plugin
Stage 2: API for intercepting URLs
This stage enabled plugins to intercept URLs added to the download queue.
Plugins are able to reject URLs from being downloaded, and change URLs and downloaded file name before adding to the queue.
Stage 3: API for intercepting downloaded files
This stage enabled plugins to intercept downloaded files.
Plugins are able to read downloaded files, and optionally scan files for additional URLs and push them into the download queue.
Stage 4: API for implementing custom HSTS, HPKP and OCSP databases.
This stage enabled plugins to custom databases for HSTS, HPKP, and OCSP.
wget2 needs to store persistent data between runs.
This is stored in flat files
respectively. This stage allows the persistent data to be loaded from arbitrary
sources and allows plugins to supply their own databases.
- Unit test
unit-tests/test-dl.cwas added for testing
tests/test-plugin.cwas added for testing all plugin-related functionality.
|File||Line coverage||function coverage|
What I learnt
Writing tests that execute in a finite time.
So far the tests I write were like, write test code to generate random input,
execute the code in question, and write an acceptor function to check for
correctness. Then run the test program under valgrind and wait patiently
for assertion failures or memory errors.
Working under wget2 taught me to write tests that cover almost all possible
cases guided by
Developing for windows.
My original plan was to offload testing on windows to others, but
during community bonding period I decided to give it a shot and attempted to
cross-compile wget2 for mingw64 and run the tests in
wine. I discovered a row of issues, fixing them essentially taught me to navigate through MSDN and find out differences in behavior of library functions in windows. Using the experience I can now port some of my own projects to work on windows.
Work mentioned in proposal but not done
- Ability for plugins to read, add and modify HTTP request and response headers