1. 01 Apr, 2019 1 commit
    • Benjamin Schubert's avatar
      lint: Fix or silence 'cyclic-import' errors and enable pylint for it · 438ed34c
      Benjamin Schubert authored
      Cyclic imports can be confusing because the order in which we import
      dependencies can make the import fail or not. We should not rely on
      ordering of imports for our code. This fixes everywhere possible the
      imports and silence explicitely some which are not convenient or would
      require big refactors
      438ed34c
  2. 14 Mar, 2019 1 commit
  3. 01 Mar, 2019 1 commit
    • Benjamin Schubert's avatar
      Let subprocess decode stdout based on locale · 5264b909
      Benjamin Schubert authored
      Subprocesses can return decoded strings if we give them a
      "universal_newlines=True" argument. We can therefore offload that to
      them, and not explicitly decode output ourselves.
      
      This also fixes multiple bugs where we would not be respecting the
      locale used by the user, and in some cases force it to "ascii".
      5264b909
  4. 27 Feb, 2019 1 commit
  5. 26 Feb, 2019 5 commits
  6. 14 Feb, 2019 2 commits
    • Jürg Billeter's avatar
      utils.py: Do not mangle absolute symlinks · 55bccf63
      Jürg Billeter authored
      Copy symlinks as they are, absolute or relative. We no longer resolve
      symlinks when copying files, which makes this safe.
      55bccf63
    • Jürg Billeter's avatar
      utils.py: Change _ensure_real_directory() to not resolve symlinks · e7e4086e
      Jürg Billeter authored
      Resolving symlinks during staging causes various issues:
      
      * Split rules may not work properly as the resolved paths will differ
        depending on whether another artifact with a directory symlink has
        been staged in the same root directory or not, e.g., as part of
        compose.
      
      * The order of symlinks in file lists is difficult to get right to
        guarantee consistent and predictable behavior as paths in a file list
        might rely on symlinks in the same file list. See #647 and #817.
      
      * Staging order differences can lead to surprising results. See #390.
      
      * Difficult to properly support absolute symlinks. Absolute symlinks are
        currently converted to relative symlinks, however, this doesn't always
        work. See #606 and #830.
      
      This will require changes in projects that rely on the current behavior.
      However, the changes are expected to be small and are often a sign of
      buggy element files. E.g., elements that don't fully obey `bindir` or
      `sbindir` variables.
      e7e4086e
  7. 13 Feb, 2019 1 commit
    • Jürg Billeter's avatar
      utils.py: Return all directories in list_relative_paths() · 925429ea
      Jürg Billeter authored
      Returning only empty directories leads to inconsistencies when computing
      a manifest by combining results from multiple list_relative_paths()
      calls as done by the compose plugin.
      
      I.e., the same directory may be empty in one dependency and non-empty in
      another dependency.  The merged file list will still contain that
      directory even though it's no longer empty.
      
      This inconsistency causes problems when calculating differences between
      manifests. Returning all directories fixes these inconsistencies.
      
      This is a change in API behavior.
      925429ea
  8. 12 Feb, 2019 1 commit
  9. 11 Feb, 2019 2 commits
    • Jürg Billeter's avatar
      utils.py: Fix sorting of symlinks to directories · d1da3fb0
      Jürg Billeter authored
      os.walk() resolves symlinks to check whether they point to a directory
      even when followlinks is set to False. We already work around that
      broken behavior by extracting symlinks from `dirnames`. However, the
      sort order was still incorrect as we returned symlinks in dirnames
      before files and other symlinks. This change fixes this, sorting all
      files and symlinks in a single list.
      d1da3fb0
    • Jürg Billeter's avatar
      utils.py: Remove list_dirs parameter from list_relative_paths() · 89973fb3
      Jürg Billeter authored
      list_dirs was always True in the BuildStream code base. There was also a
      bug in the list_dirs=False code path as it did not return symlinks in
      `dirnames`.
      
      This is an API break, however, there are no known external callers.
      89973fb3
  10. 24 Jan, 2019 1 commit
  11. 18 Jan, 2019 1 commit
    • Tristan Van Berkom's avatar
      utils.py: Added _tempnamedfile() · 99699ffc
      Tristan Van Berkom authored
      When used in a child process in BuildStream, this should be
      used instead of tempfile.NamedTemporaryFile() directly, otherwise
      we fail to cleanup the file on SIGTERM.
      99699ffc
  12. 16 Jan, 2019 1 commit
    • Tristan Van Berkom's avatar
      element.py: Cleanup temporary staging directories on termination · 19adc9e5
      Tristan Van Berkom authored
      Use utils._tempdir() which in turn uses _signals.terminator() for
      this purpose.
      
      This also changes utils._tempdir() to use utils._force_rmtree()
      so that it is safe to use for cleaning up the staging directory.
      
      This patch fixes orphaned temporary directories being left behind
      in the artifact cache directory at forceful termination time.
      19adc9e5
  13. 07 Jan, 2019 1 commit
    • Tristan Van Berkom's avatar
      Fix stack traces discovered with ^C forceful termination. · 891fcb0e
      Tristan Van Berkom authored
        * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
      
          These are caused because we issue SIGTERM, and if the process
          has not exited after a timeout, we kill it.
      
        * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
          redundantly, they are already ignored.
      
      It seems that we were ignoring it after sleeping when terminating
      tasks from the scheduler... but we were not ignoring it when performing
      the same pattern in the `Plugin.call()` -> `utils._call()` path, so
      we were still getting these exceptions at termination time from host
      tool processes launched by source plugins.
      891fcb0e
  14. 20 Dec, 2018 1 commit
    • Chandan Singh's avatar
      buildstream/utils.py: Fix regex Deprecation Warning · e8055a56
      Chandan Singh authored
      Specify flags at the start of the expression as per the recommendation
      of the standard library.
      
      Without this patch, we currently get the following warning:
      
      ```
      tests/examples/junctions.py::test_open_cross_junction_workspace
        /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the expression '\\/[^/]*\\Z(?ms)'
          regexer = re.compile(expression)
      ```
      e8055a56
  15. 11 Dec, 2018 1 commit
  16. 05 Dec, 2018 1 commit
  17. 21 Nov, 2018 2 commits
  18. 19 Nov, 2018 1 commit
    • Benjamin Schubert's avatar
      Extract atomic move function to utils.py · 88089d2d
      Benjamin Schubert authored
      Moving atomically a file/directory can be tricky since different
      errors might be raised for the same underlying problem.
      
      Having a utility function to reduce this discrepancies will help
      in ensuring we have correct behavior
      88089d2d
  19. 06 Nov, 2018 1 commit
  20. 26 Oct, 2018 1 commit
  21. 25 Oct, 2018 2 commits
  22. 16 Oct, 2018 1 commit
  23. 04 Oct, 2018 1 commit
    • Valentin David's avatar
      Catch correct exception from shutil.rmtree · a7984218
      Valentin David authored
      Python documentation is not clear on what shutil.rmtree is supposed to
      raise. However from the source code, it is obvious that OSError are
      raised, but never shutil.Error.
      
      It is not easy to test in normal condition because issues happen
      usually in combination with a FUSE filesystem, a probably a race
      condition where `fstatat` returns an error just before the filesystem
      being unmounted.
      
      Fixes #153.
      a7984218
  24. 03 Oct, 2018 2 commits
  25. 27 Sep, 2018 1 commit
    • Phillip Smyth's avatar
      utils.py: Reworked safe_remove · b78ae767
      Phillip Smyth authored
      Non-Linux platforms don't return EISDIR when attempting to unlink a
      directory.
      
      Stopped safe_remove attempting to unlink dirs
      Previously safe_remove would attempt to unlink a path
      Before attempting to remove it if it was a dir
      
      Now it checks for a dir before that step
      b78ae767
  26. 14 Sep, 2018 1 commit
  27. 29 Aug, 2018 1 commit
  28. 28 Aug, 2018 1 commit
  29. 22 Aug, 2018 1 commit
    • Will Salmon's avatar
      Catch Non Numeric versions · eeaddbac
      Will Salmon authored
      This patch just displays a better message than the default stack trace
      but dose not try to fix the problem. A further patch will be created but
      it effects versioneer so may take longer to land as it may need to go
      via versioneer mainline.
      eeaddbac
  30. 12 Aug, 2018 1 commit
  31. 01 Aug, 2018 1 commit