BuildStream chokes when staging tarfiles with hardlinks that point outside of a chosen base directory
Summary
When BuildStream stages a tarfile, it will remove the first couple of characters from certain file names to extract things correctly. This is all fine and dandy, however, the code as-is, fails to extract certain kinds of hardlinks, as duly documented in the plugin itself.
Steps to reproduce
- Create a tarfile with a hardlink that points outside its base directory.
- Stage it.
What is the current bug behavior?
BuildStream reports a BUG and stops building.
What is the expected correct behavior?
BuildStream should continue building!
Relevant logs and/or screenshots
py37 develop-inst-noop: /home/tlater/Documents/Work/buildstream/buildstream
py37 installed: apipkg==1.5,arpy==1.1.1,astroid==2.1.0,atomicwrites==1.3.0,attrs==18.2.0,-e git+git@gitlab.com:BuildStream/buildstream.git@f61f47cd519fa2402ff0b7d7e4a91ce63b84b0c3#egg=BuildStream,Click==7.0,coverage==4.4,Cython==0.29.7,execnet==1.5.0,grpcio==1.17.1,isort==4.3.9,Jinja2==2.10,lazy-object-proxy==1.3.1,MarkupSafe==1.1.0,mccabe==0.6.1,more-itertools==6.0.0,pathlib2==2.3.3,pluggy==0.9.0,pluginbase==0.7,protobuf==3.6.1,psutil==5.4.8,py==1.8.0,pycodestyle==2.5.0,pyftpdlib==1.5.4,pylint==2.2.2,pyroaring==0.2.6,pytest==4.3.0,pytest-cov==2.6.1,pytest-datafiles==2.0,pytest-env==0.6.2,pytest-forked==1.0.2,pytest-timeout==1.3.3,pytest-xdist==1.26.1,ruamel.yaml==0.15.51,six==1.12.0,typed-ast==1.3.1,ujson==1.35,wrapt==1.11.1
py37 run-test-pre: PYTHONHASHSEED='2994009352'
py37 run-test: commands[0] | pytest --basetemp /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp --cov=buildstream --cov-config .coveragerc --integration -m integration --no-cov -x --lf
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-4.3.0, py-1.8.0, pluggy-0.9.0 -- /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/bin/python
cachedir: .tox/py37/.pytest_cache
rootdir: /home/tlater/Documents/Work/buildstream/buildstream, inifile: setup.cfg
plugins: xdist-1.26.1, timeout-1.3.3, forked-1.0.2, env-0.6.2, datafiles-2.0, cov-2.6.1
collecting ... collected 1292 items / 1291 deselected / 1 selected
run-last-failure: rerun previous 2 failures
tests/examples/autotools.py::test_autotools_build FAILED [100%]
=================================== FAILURES ===================================
_____________________________ test_autotools_build _____________________________
cli = <buildstream.testing.runcli.CliIntegration object at 0x7f9b78f73828>
datafiles = local('/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp/test_autotools_build0')
@pytest.mark.skipif(MACHINE_ARCH != 'x86-64',
reason='Examples are written for x86-64')
@pytest.mark.skipif(not IS_LINUX or not HAVE_BWRAP, reason='Only available on linux with bubblewrap')
@pytest.mark.datafiles(DATA_DIR)
def test_autotools_build(cli, datafiles):
project = str(datafiles)
checkout = os.path.join(cli.directory, 'checkout')
# Check that the project can be built correctly.
result = cli.run(project=project, args=['build', 'hello.bst'])
> result.assert_success()
tests/examples/autotools.py:29:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <buildstream.testing.runcli.Result object at 0x7f9b78f7dda0>
fail_message = ''
def assert_success(self, fail_message=''):
> assert self.exit_code == 0, fail_message
E AssertionError
src/buildstream/testing/runcli.py:115: AssertionError
----------------------------- Captured stdout call -----------------------------
BuildStream exited with code -1 for invocation:
bst --no-colors --config /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp/test_autotools_build0/cache/buildstream.conf --directory /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp/test_autotools_build0 build hello.bst
Program stderr was:
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.3.0+2355.gf61f47cd.dirty
Session Start: Monday, 24-06-2019 at 13:40:01
Project: autotools (/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp/test_autotools_build0)
Targets: hello.bst
Cache Usage: 12K / None (0%)
User Configuration
Configuration File: /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp/test_autotools_build0/cache/buildstream.conf
Cache Directory: /home/tlater/Documents/Work/buildstream/buildstream/integration-cache/cache-9kfotmgf
Log Files: /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/tmp/test_autotools_build0/cache/logs
Source Mirrors: /home/tlater/Documents/Work/buildstream/buildstream/integration-cache/sources
Build Area: /home/tlater/Documents/Work/buildstream/buildstream/integration-cache/cache-9kfotmgf/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
buildable 7ea72715664f3125eac3063c5120c47ffe4327f90b61af0d809b6f153f617b49 base/freedesktop.bst
waiting d098691a4af90e544427c18b3c051a738fc81558cb7b664287a43fa82a0fff61 base.bst
waiting b4cda6988d11b1606e330a55aad1d2e9f1468b623e448bd09ae7e931f3e60d81 hello.bst
===============================================================================
[--:--:--][7ea72715][ fetch:base/freedesktop.bst ] START autotools/base-freedesktop/7ea72715-fetch.5780.log
[--:--:--][b4cda698][ fetch:hello.bst ] START autotools/hello/b4cda698-fetch.5782.log
[00:00:00][b4cda698][ fetch:hello.bst ] SUCCESS autotools/hello/b4cda698-fetch.5782.log
[00:00:18][7ea72715][ fetch:base/freedesktop.bst ] BUG Fetch
An unhandled exception occured:
Traceback (most recent call last):
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_scheduler/jobs/job.py", line 699, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 108, in child_process
return self._action_cb(self._element)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 45, in process
element._fetch(fetch_original=self._fetch_original)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/element.py", line 2183, in _fetch
self.__cache_sources()
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/element.py", line 2932, in __cache_sources
self.__sourcecache.commit(source, [])
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_sourcecache.py", line 160, in commit
source._stage(tmpdir)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/source.py", line 740, in _stage
self.stage(directory)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/plugins/sources/tar.py", line 117, in stage
tar.extractall(path=directory, members=self._extract_members(tar, base_dir))
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2002, in extractall
numeric_owner=numeric_owner)
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2044, in extract
numeric_owner=numeric_owner)
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2122, in _extract_member
self.makelink(tarinfo, targetpath)
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2210, in makelink
self._extract_member(self._find_link_target(tarinfo),
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2380, in _find_link_target
raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'c/ssl/ct_log_list.cnf' not found"
[00:00:18][ ][ main:core activity ] FAILURE Build
Failure Summary
base/freedesktop.bst:
[00:00:18][7ea72715][ fetch:base/freedesktop.bst ] BUG Fetch
An unhandled exception occured:
Traceback (most recent call last):
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_scheduler/jobs/job.py", line 699, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 108, in child_process
return self._action_cb(self._element)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 45, in process
element._fetch(fetch_original=self._fetch_original)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/element.py", line 2183, in _fetch
self.__cache_sources()
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/element.py", line 2932, in __cache_sources
self.__sourcecache.commit(source, [])
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/_sourcecache.py", line 160, in commit
source._stage(tmpdir)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/source.py", line 740, in _stage
self.stage(directory)
File "/home/tlater/Documents/Work/buildstream/buildstream/src/buildstream/plugins/sources/tar.py", line 117, in stage
tar.extractall(path=directory, members=self._extract_members(tar, base_dir))
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2002, in extractall
numeric_owner=numeric_owner)
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2044, in extract
numeric_owner=numeric_owner)
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2122, in _extract_member
self.makelink(tarinfo, targetpath)
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2210, in makelink
self._extract_member(self._find_link_target(tarinfo),
File "/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/tarfile.py", line 2380, in _find_link_target
raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'c/ssl/ct_log_list.cnf' not found"
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 1, skipped 1, failed 1
Build Queue: processed 0, skipped 0, failed 0
=============================== warnings summary ===============================
.tox/py37/lib/python3.7/site-packages/pluginbase.py:439
/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/site-packages/pluginbase.py:439: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
fromlist, level)
-- Docs: https://docs.pytest.org/en/latest/warnings.html
WARNING: Coverage disabled via --no-cov switch!
========================== slowest 20 test durations ===========================
18.60s call tests/examples/autotools.py::test_autotools_build
0.06s teardown tests/examples/autotools.py::test_autotools_build
(0.00 durations hidden. Use -vv to show these durations.)
=========================== warnings summary (final) ===========================
.tox/py37/lib/python3.7/site-packages/pytest_cov/plugin.py:253
/home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/lib/python3.7/site-packages/pytest_cov/plugin.py:253: PytestWarning: Coverage disabled via --no-cov switch!
warnings.warn(pytest.PytestWarning(message))
-- Docs: https://docs.pytest.org/en/latest/warnings.html
============ 1 failed, 1291 deselected, 2 warnings in 25.13 seconds ============
ERROR: InvocationError for command /home/tlater/Documents/Work/buildstream/buildstream/.tox/py37/bin/pytest --basetemp .tox/py37/tmp --cov=buildstream --cov-config .coveragerc --integration -m integration --no-cov -x --lf (exited with code 1)
___________________________________ summary ____________________________________
ERROR: py37: commands failedOther relevant information
- BuildStream version affected: /milestone %BuildStream_v1.2 BuildStream_v2.0
Edited by Tristan Maat