Skip to content

Add support to save file extended attributes

Didik Setiawan requested to merge dstw/wget2:add-xattr into master
  • configure.ac: Check for xattr availability
  • src/Makefile.am: Add wget_xattr.h
  • src/options.c: Add new config 'xattr', (parse_execute): Add new option 'xattr
  • src/wget.c: (_prepare_file): Add condition for 'xattr', (http_send_request): Add parameter 'original_url', (add_url): Assign 'original_url' with proper value, (download_thread): Add variable 'original_url', (write_xattr_metadata): New function, (set_file_metadata): New function
  • src/wget_job.h: Add variable 'original_url'
  • src/wget_options.h: Add new config 'xattr'
  • src/wget_xattr.h: Add new function declaration 'set_file_metadata', Add fsetxattr() if defined in operating system

These attributes provide a lightweight method of later determining where a file was downloaded from. Adapted from Wget 1.x.

This patch changes:

  • autoconf detects whether extended attributes are available and enables the code if they are.
  • The new flags --xattr and --xattr=no control whether xattr is enabled.
  • The original and redirected URLs are recorded as shown below.
  • This works for both single fetches and recursive mode.

The attributes that are set are:
user.xdg.origin.url: The URL that the content was fetched from.
user.xdg.referrer.url: The URL that was originally requested.
user.mime_type: Two-part identifier for file formats and format contents.
user.charset: File encoding.

For example, where http://lwn.net redirects to https://lwn.net:
$ wget2 --xattr http://lwn.net
...
$ getfattr -d index.html
user.charset:"UTF-8"
user.mime_type:="text/html"
user.xdg.origin.url="https://lwn.net/"
user.xdg.referrer.url="http://lwn.net/"

TODO:

  • Add test case
  • Add docs

Still far from complete. Here some minus I found:

  • At src/wget.c, when calling set_file_metadata, I still don't know how to pass value of URI.
  • When built, it break ./test/test-wget-1.

Based on #210 (closed)

Edited by Didik Setiawan

Merge request reports