doc: Adding new architecture document describing the data model

doc/source/arch_data_model.rst

+
+
+Data model
+==========
+This section details the data model on which the BuildStream core operates. This
+includes an overview of the project data model which is BuildStream's main input,
+the user preferences, and local state.
+
+
+Project
+-------
+The ``Project`` object is the main component of a given BuildStream *project*, and
+is responsible for loading and validating the :ref:`project.conf <projectconf>`, and
+providing this loaded *project data* in a convenient way to the BuildStream core.
+
+Conceptually, the *project* is a container for the :mod:`Elements <buildstream.element>`,
+which are declared within a user's project, and as such acts as a factory for instantiating
+elements at load time.
+
+
+Element
+-------
+:mod:`Elements <buildstream.element>` are the main processing unit in a pipeline. These
+are the loaded representation of the ``.bst`` files loaded from the :ref:`project's element path
+<project_element_path>`.
+
+The *Element* is an abstract base class which cannot do anything on its own, its
+concrete class is defined by *plugins* which are either included in the BuildStream
+:ref:`core set of plugins <plugins>` or loaded from external sources :ref:`defined by the project
+<project_plugins>`
+
+The responsibilities of an element include:
+
+* Loading the element's configuration from the core provided dictionary
+* Providing a unique key for any element specific configuration which might
+  effect the output produced by the element
+* Configuring the sandbox
+* Staging the data into the sandbox, which might include Sources and
+  the outputs of previous elements
+* Assembling the output *artifact*
+
+
+Element data structure
+~~~~~~~~~~~~~~~~~~~~~~
+The properties of an element are a composition of what the BuildStream core understands,
+the configurations exposed by the Element plugin, and free form data which allows
+annotations and configurations which can be read back by reverse dependencies during
+processing, as illustrated here:
+
+.. image:: images/arch-datamodel-element.svg
+   :align: center
+
+
+Element composition
+~~~~~~~~~~~~~~~~~~~
+The element is composed of configurations which are sourced from various entry
+points using the low level YAML utilities.
+
+This composition takes place after :ref:`includes <format_directives_include>` and
+:ref:`conditional <format_directives_conditional>` directives are processed, while
+:ref:`list composition <format_directives_list_prepend>` directives are processed
+as a result of this composition.
+
+Here is a diagram showing which sources take precedence in the composition process
+which results in the final element configuration being resolved:
+
+.. image:: images/arch-datamodel-element-composition.svg
+   :align: center
+
+Note that not all *BuildStream Core Data* is understood by the *Element*, but a great
+deal of configurations understood by the *Element* is also understood by the core and
+has default configurations built into BuildStream and configurable with the project
+configuration. These include values such as *variables*, *environment*, *sandbox*, etc.
+
+As shown above, composition is performed in two stages, as we only need to composite
+the data from the toplevel element declaration against the composition of previous
+stages every time we instantiate an element.
+
+
+Source
+------
+:mod:`Sources <buildstream.element>` are the abstract objects which are responsible
+for obtaining remote source code or data to import into the build environment, and
+ensuring that it is done in a bit-for-bit reproducible way without any contamination
+of the host or build environment.
+
+This is to say that:
+
+* User configuration on the host, or filesystem outside of BuildStream designated
+  directories, must never be modified as a side effect of running BuildStream.
+
+* When the Source uses host tools, host side configurations must never result in
+  deviations of what is staged to a build directory. The Source must behave exactly
+  the same way regardless of host side configurations.
+
+The responsibilities of a source include:
+
+* Loading the source's configuration from the core provided dictionary
+* Providing a unique key for any source specific configuration which might
+  effect the staged source
+* Implement discovery of new versions of the source upstream (referred to as *"tracking"*)
+* Staging the unpacked source to a given directory
+* Preparing workspaces
+
+
+Source data structure
+~~~~~~~~~~~~~~~~~~~~~
+Similar to the *Element*, the properties of a source are a composition of what
+the BuildStream core understands and the configurations exposed by the Source
+plugin:
+
+.. image:: images/arch-datamodel-source.svg
+   :align: center
+
+.. note::
+
+   In .bst files, the BuildStream core configurations and Source specific configurations
+   share the same dictionary.
+
+   Strictly speaking this is limiting, but provides a measure of convenience as .bst
+   files are a bit less wordy to express.
+
+
+Source composition
+~~~~~~~~~~~~~~~~~~
+Source composition is much simpler than Element composition, because defaults
+cannot be specified at the project level, excepting for Source type specific
+value overrides.
+
+.. image:: images/arch-datamodel-source-composition.svg
+   :align: center
+
+
+Context
+-------
+The Context object is a very centric part of the BuildStream data model, and is
+not a part of the Project data described above but rather is where we load and
+store all of the user preferences.
+
+User preferences are sourced from various locations, but usually have a default,
+an option in the user configuration file, and an option to override it on the
+command line.
+
+.. image:: images/arch-datamodel-context.svg
+   :align: center
+
+Asides from being a focal point for loading and storing all user configuration,
+the Context object also plays a central role in the logging framework.
+
+
+Workspaces
+----------
+The Workspaces object is yet another kind of state. Unlike the Context and
+the Project data model, the Workspaces object loads, saves and stores in
+memory the local state regarding a user's active and open workspaces.
+
+These are stored in the local state ``.bst/`` subdirectory of users projects.

doc/source/main_architecture.rst

@@ -9,3 +9,4 @@ This section provides details on the overall BuildStream architecture.
    :maxdepth: 2
 
    arch_overview
+   arch_data_model