1. 03 Jul, 2020 2 commits
  2. 02 Jul, 2020 1 commit
  3. 30 Jun, 2020 3 commits
  4. 25 Jun, 2020 2 commits
  5. 24 Jun, 2020 6 commits
    • bst-marge-bot's avatar
      Merge branch 'tristan/junction-jungle' into 'master' · 88f280d5
      bst-marge-bot authored
      Explicit control of junction configuration overrides
      
      Closes #1326
      
      See merge request !1901
      88f280d5
    • Tristan van Berkom's avatar
      docs: Updating documentation regarding junctions and conflicts. · 3d5d655e
      Tristan van Berkom authored
      This commit:
      
        * Redocuments the `junction.py` element almost in it's entirity
      
        * Describes the `overrides` feature
      
        * Describes how to use `link` elements to derive junction configuration
          from subprojects.
      
        * Adds project.conf documentation for junction `duplicates`
      
        * Adds project.conf documentation for junction `internal`
      3d5d655e
    • Tristan van Berkom's avatar
    • Tristan van Berkom's avatar
      _project.py: Implement internal junctions · 05f6958f
      Tristan van Berkom authored
      This commit adds support for the `internal` list of the project's
      `junctions` configuration section.
      
      _loader/loadcontext.py is updated with this commit to consider
      the internal markers and avoid raising errors for collisions with
      internal junctions, as well as being enhanced to list internal
      junctions in any conflicting junction error messages.
      05f6958f
    • Tristan van Berkom's avatar
    • Tristan van Berkom's avatar
      _loader: Add support for duplicate junctions · 9da47db4
      Tristan van Berkom authored
      This adds some measure of tolerance to duplicate projects by
      accepting cases where multiple projects are loaded but whitelisted
      to be loaded as duplicates by the project.conf.
      
      This change does the following:
      
        * _project.py: Add `junction_is_duplicated()` API and parse `duplicates`
      
          The project now maintains a table of junctions which are considered
          "duplicated" per project, and exposes an API for the loader to query
          whether a given loader is explicitly listed as a duplicate (allowing
          multiple "duplicated" instances of that project to coexist in the same
          pipeline).
      
          This new API searches for the passed Loader instance by way of it's
          project relative search paths specified in the duplicates. Using the
          `Loader.get_loader()` codepath ensures that any links get resolved
          and that any overrides along the way will be considered correctly
          while attempting to match the duplicate.
      
        * _loader/loader.py: Few changes here.
      
          - Allow Loader.get_loader() to operate in a mode which does not
            load subprojects.
      
          - Add new function Loader.ancestors(), this is a generator which yields
            every ancestor loader which is capable of addressing the given loader,
            including ancestors for which their junction definitions to this loader
            have been overridden.
      
            To this end, we now record `_alternative_parents` when searching for
            overrides.
      
          - Now implements __str__() in order to print an identifying string
            for a loader in error messages
      
        * _loader/loadcontext.py: Enhanced to now ask projects whether they are
          marked as duplicates in order to avoid erroring out on project collisions
          if those are explicitly marked to not be an error.
      
          We now also split out the new function LoadContext.assert_loaders(), which
          is to be called once at the end of loading the whole pipeline recursively.
      
          This reduces the complexity of searching the loaded graph while loading
          the graph, and also reports more complete errors (all instances of a
          conflicting project will be reported in the error at once).
      9da47db4
  6. 23 Jun, 2020 2 commits
    • Tristan van Berkom's avatar
      tests/format/junctions.py: Adding tests for the overrides feature · 364ea9dd
      Tristan van Berkom authored
        * Test various scenarios of overriding junctions, including
          deep paths as junctions to override, and as junctions to use
          to override.
      
        * Test conflicting junction configurations, ensuring that we
          report both provenances of where the junctions were declared.
      
        * Test circular references in element paths while declaring overrides,
          for instance when trying to override a subproject using a deeper
          definition of the same subproject.
      364ea9dd
    • Tristan van Berkom's avatar
      junctions: Replace coalescing with explicit overrides · 5f75fefe
      Tristan van Berkom authored
      This patch removes the functionality of coalescing junction
      configurations on junction element names, and replaces it with
      an explicit format which allows junction declarations to override
      junctions in subprojects with junction declarations from the local
      project.
      
      Changes:
      
        * plugins/elements/junction.py: Load the overrides dictionary
      
        * exceptions.py: Added new CIRCULAR_REFERENCE to LoadErrorReason.
      
        * _loader/loadcontext.py: Add new register_loader() function which
          is called by the Loader to register all loaders with the context,
          delegating the task of detecting conflicting junctions to the
          load context.
      
        * _loader/loader.py: When loading a junction, check if there is
          an override from the parent project and use that instead.
      
          Register with the LoadContext, and pass the LoadContext
          and provenance along when creating a Project
      
          Further, we now protect against circular element path references
          with a new _loader_search_provenances table.
      
        * _project.py: Pass new `provenance` member through the constructor
          and onwards into the child loader.
      5f75fefe
  7. 19 Jun, 2020 4 commits
    • Tristan van Berkom's avatar
    • Tristan van Berkom's avatar
    • Tristan van Berkom's avatar
      tests/format/junctions.py: Major refactor · a4a9b5f2
      Tristan van Berkom authored
      This commit refactors the junctions test to use more parameterization
      and to remove copy_subproject(), using statically committed data as
      much as possible.
      
      This is because copy_subproject() causes data to be shared among tests
      in such a way that when such data get's modified, it easily causes
      unintended side effects on adjacent test cases, better to keep this
      data separate.
      
      Overview of changes:
      
        * Remove copy_subproject()
      
        * Split up test data into case specific directories, sometimes
          reusing a directory among various related tests
      
        * Use @pytest.mark.parameterize() as much as possible for better
          coverage and more clearly expressed test cases
      
        * Adds update_project() to modify a project.conf inline, as is
          done in some other tests like tests/plugins/loading.py
      
        * Removes tests related to junction name coalescing, this feature
          will be removed later in this branch and other tests related
          to junction overrides will replace these.
      
        * Removes some redundant tests
      
        * Removes a comment about how junction name coalescing can cause
          errors when trying to open a junction but the project.conf is
          missing. We continue to test missing project.conf files in a
          variety of scenarios, but the comment will be rendered irrelevant
          with the removal of junction name coalescing.
      
        * Change the git related tests to use tar instead, this serves
          the same purpose, but tar will remain a core plugin in BuildStream 2.
      a4a9b5f2
    • Tristan van Berkom's avatar
      _loader: Fix invalid junction error to include provenance · ed027578
      Tristan van Berkom authored
      When a junction element has dependencies, we were not even specifying
      which junction element was guilty of having dependencies and only
      saying that such a thing would be forbidden (wherever it might be in
      your project).
      
      Fix this by using the provenance of the dependency in the junction,
      specifying exactly where the invalid dependency exists.
      
      This required carrying the provenance onto the other Dependency() object.
      ed027578
  8. 18 Jun, 2020 1 commit
  9. 17 Jun, 2020 2 commits
  10. 16 Jun, 2020 3 commits
    • bst-marge-bot's avatar
      Merge branch 'tristan/load-context' into 'master' · 05ecd800
      bst-marge-bot authored
      _loader: Adding LoadContext
      
      See merge request !1964
      05ecd800
    • Tristan van Berkom's avatar
      _loader: Adding LoadContext · deb65627
      Tristan van Berkom authored
      Instead of passing around many details though calling signatures
      throughout the loader code, create a single LoadContext object
      which holds any overall loading state along with any values which
      are constant to a full load process.
      
      Overall this patch does:
      
        * _frontend/app.py: No need to pass Stream.fetch_subprojects() along anymore
      
        * _loader/loadelement.pyx: collect_element_no_deps() no longer takes a task argument
      
        * _loader/loader.py: Now the Loader has a `load_context` member, and no more
          `_fetch_subprojects` member or `_context` members
      
          Further, `rewritable` and `ticker` is no longer passed along through all
          of the recursing calling signatures, and `ticker` itself is finally removed
          because this has been replaced a long time ago with `Task` API from `State`.
      
        * _pipeline.py: The load() function no longer has a `rewritable` parameter
      
        * _project.py: The Project() is responsible for creating the toplevel
          LoadContext() if one doesn't exist yet, and this is passed through
          to the Loader() (and also passed to the Project() constructor by the
          Loader() when instantiating subprojects).
      
        * _stream.py: The `Stream._fetch_subprojects()` is now private and set
          on the project when giving the Project to the Stream in `Stream.set_project()`,
          also the Stream() sets the `rewritable` state on the `LoadContext` at the
          earliest opportunity, as the `Stream()` is the one who decides this detail.
      
          Further, some double underscore private functions are now regular single
          underscores, there was no reason for this inconsistency.
      
        * tests/internals/loader.py: Updated for API change
      deb65627
    • Tristan Van Berkom's avatar
      Merge branch 'tristan/nuke-pickle-jobber' into 'master' · 3774ce42
      Tristan Van Berkom authored
      Completely abolish job pickling.
      
      See merge request !1965
      3774ce42
  11. 15 Jun, 2020 1 commit
  12. 12 Jun, 2020 2 commits
  13. 11 Jun, 2020 5 commits
    • Tristan van Berkom's avatar
      _project.py: Reorganize methods · 2eccea85
      Tristan van Berkom authored
      Add some comments clearly separating public and private methods,
      and moving the _validate_node() private method into the private
      section, renaming it _validate_toplevel_node() for better clarity.
      2eccea85
    • Tristan van Berkom's avatar
      _project.py: Remove invoked_from_workspace_element() · b29172e0
      Tristan van Berkom authored
      This is no longer needed.
      b29172e0
    • Tristan van Berkom's avatar
      _stream.py: Remove workspace_is_required() · d5121cc1
      Tristan van Berkom authored
      This API was only used to issue a not very useful message in the CLI
      when removing a workspace the user happened to have invoked BuildStream
      from.
      
      The API also didn't really make any sense, being documented as:
      
        "Checks whether the workspace belonging to element_name is
         required to load the project"
      
      This in fact meant whether having that workspace open was required
      for BuildStream to be invoked, due to the metadata encoded into the
      workspace directory.
      d5121cc1
    • Tristan van Berkom's avatar
      _frontent/cli.py: Removed message to user informing the workspace is closed. · 618173c6
      Tristan van Berkom authored
      When closing a workspace while the `workspace close` command was issued
      from the closed workspace, we go to great lengths to inform the user
      that they can no longer run bst commands from the closed workspace directory.
      
      This is not worthwhile, the user will know this the next time they run
      `bst` anyway.
      618173c6
    • Tristan van Berkom's avatar
      tests/frontend/workspace.py: Removing tests · f72aa203
      Tristan van Berkom authored
      Remove tests which check for a user message to be issued upon closing
      a workspace who's metadata was used to launch BuildStream and find
      the BuildStream project directory.
      f72aa203
  14. 10 Jun, 2020 6 commits