Downloadable file source is not setting a user agent; rejected by gitlab

Summary

When using the downloadable file source to fetch a source archive from gitlab.com, BuildStream exits with an error. Gitlab rejects our request with a "403 Forbidden" error. It appears this is a recent new behaviour, and it appears to stem from BuildStream's use of the default user agent with urllib.request. Using a different user agent, gitlab.com allows us to download the file normally.

It would be good practice to set a user agent, which will help to differentiate BuildStream from other scripts downloading files.

Steps to reproduce

Create a buildstream project with the following elements...

gitlab-tar-file.bst:

kind: manual

build-depends:
- base.bst

sources:
- kind: tar
  url: https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz

bst.bst:

kind: import
description: |

    Alpine Linux base runtime

sources:
- kind: tar

  # This is a post doctored, trimmed down system image
  # of the Alpine linux distribution.
  #
  url: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639

In your project, run bst track gitlab-tar-file.bst.

What is the current bug behavior?

BuildStream exits with the following output:

[--:--:--][????????][track:gitlab-tar-file.bst           ] START   gitlab-tar-issue/gitlab-tar-file/????????-track.148575.log
[--:--:--][????????][track:gitlab-tar-file.bst           ] START   Tracking https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz
[00:00:00][????????][track:gitlab-tar-file.bst           ] FAILURE Tracking https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz
[00:00:00][????????][track:gitlab-tar-file.bst           ] FAILURE tar source at gitlab-tar-file.bst [line 7 column 2]: Error mirroring https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz: HTTP Error 403: Forbidden

    Printing the last 20 lines from log file:
    /home/dylan/.cache/buildstream/logs/gitlab-tar-issue/gitlab-tar-file/????????-track.148575.log
    ======================================================================
    [--:--:--] START   gitlab-tar-file.bst: Track
    [--:--:--] LOG     gitlab-tar-file.bst: Build environment for element gitlab-tar-file.bst

        HOME: /tmp
        LC_ALL: C
        LOGNAME: tomjon
        MAKEFLAGS: -j8
        PATH: /usr/bin:/bin:/usr/sbin:/sbin
        SHELL: /bin/sh
        SOURCE_DATE_EPOCH: '1320937200'
        TERM: dumb
        TZ: UTC
        USER: tomjon
        USERNAME: tomjon
        V: '1'
    [--:--:--] START   gitlab-tar-file.bst-0: Tracking https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz
    [00:00:00] FAILURE gitlab-tar-file.bst-0: Tracking https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz
    [00:00:00] FAILURE gitlab-tar-file.bst: tar source at gitlab-tar-file.bst [line 7 column 2]: Error mirroring https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz: HTTP Error 403: Forbidden
    ======================================================================

Possible fixes

Gitlab appears to be rejecting an arbitrary list of user agents, one of which is the default used by urllib (Python-urllib/3.7). so a good fix here would be to set our own over here: https://gitlab.com/BuildStream/buildstream/-/blob/1.4.2/buildstream/plugins/sources/_downloadablefilesource.p

Other relevant information

We can also reproduce this behaviour using a simple Python program which behaves the same as plugins/sources/_downloadablefilesource.py:

from urllib.request import Request, urlopen
from urllib.error import HTTPError
url = 'https://gitlab.com/BuildStream/buildstream/-/archive/1.4.1/buildstream-1.4.1.tar.gz'
request = Request(url)
request.add_header('Accept', '*/*')
# This program works correctly if we uncomment the following line:
# request.add_header('User-Agent', 'BuildStream (+http://buildstream.build)')
try:
    response = urlopen(request)
except HTTPError as err:
    print(err.getcode(), err.reason)
    print(err.info())
Edited by Dylan McCall