1. 30 Nov, 2018 20 commits
    • Lachlan's avatar
      Add script to update run token · a99abac5
      Lachlan authored
      a99abac5
    • Lachlan's avatar
      Add generation of buildstream configuration file · 631c0172
      Lachlan authored
      * New file - generate_buildstream_config.py
         - Acts as pricinciple orchestration point for configuring
           benchmarking runs.
         - Checks the tenability of settings to make sure that cached result datasets
           will be coherent if the benchmarking results to be generated are
           added.
         - Checks if the previous run settings do not match the cached results
      631c0172
    • Lachlan's avatar
      Add ad-hoc test for token processing · 3d1d08d0
      Lachlan authored
      * The token file needs to be modified for consistency with the
        developers arrangements during testing.
      3d1d08d0
    • Lachlan's avatar
      Add token creation, parsing and validation · e23222a9
      Lachlan authored
      * New file - token_file_processing.py
        - Adds functionality to process, generate and validate token files.
        - Token files are used to confirm if new benchmarking runs are
          required to confirm what (if any) Buildstream commits need testing
          given what was previously tested.
        - Token files also allow checks to be made as to whether the currently
          cached data sets are coherent and whether aborted runs have been
          included.
      e23222a9
    • Lachlan's avatar
      Add functionatlity to get sha of commits · b2c1b631
      Lachlan authored
      * New file - get_sha_commits.py
      
        * Add functionality to get the commits between two given sha points
          for a particular repo, branch, previously processed commit and last
          commit.
      
        * If the two points are the same, then return just the last commit/
      b2c1b631
    • Lachlan's avatar
      Add generation of benchmark configuration · 418d15d6
      Lachlan authored
      * New File - generate_benchmark_config.py
        - Adds function to generate a benchmark configuration file
          that will individually test a number of individual commit
          points on a given branch.
        - The configuation file is generated from a templated version
          and a list of SHAs, together with the branch they are associated
          with.
        - Docker version is configurable, but defaulted to latest for
          the moment.
      418d15d6
    • Lachlan's avatar
      Add template version of last benchmarking token file · 73c4591d
      Lachlan authored
      * Last benchmark token file will be stored with generated results
        to confirm what the last buildstream revision used for benchmarking
        was.
      
      * This allows for all shas to be determined between the last
        benchmarking run and the current master.
      
      * The expectation is that each sha will be used as a base for running a
        particular benchmark so that each merge request to buildstream master
        can be characterised.
      
      * The benchmarking branch and sha are included for further consistency
        checks.
      73c4591d
    • Lachlan's avatar
      Add template form of generic benchmark extension · e63a1aaf
      Lachlan authored
      * Benchmarking allows for other versions to be tested by including
        a "addon-version.benchmark" type file. This template file is
        used to generate addon configuration for each buildstream commit
        which needs to be tested.
      e63a1aaf
    • Lachlan's avatar
      Add benchmarking git sha to results · e5169ca4
      Lachlan authored
      e5169ca4
    • Lachlan's avatar
    • Lachlan's avatar
      Add Pipeline cache directory as artifact element · 8555ac9e
      Lachlan authored
      8555ac9e
    • Lachlan's avatar
      Make staging of jobs explicit · f093ce92
      Lachlan authored
      * Previously it seems that benchmarking jobs could potentially be
        done in parallel which might impact upon results. This forces
        the benchmarking tests to be done in series.
      f093ce92
    • Lachlan's avatar
      Add explicit line for web page output taking into account job ID · b9c40da4
      Lachlan authored
      * WEB page output path defined and is based per job.
      
      * Make output web directorys if it does not already exist.
      
      * Generate web pages
      
      * Copy resulting pages to pipeline cache for unwrapping later.
      
      * Make pipeline cache dependent on master branch being triggered
      b9c40da4
    • Lachlan's avatar
      ae30e5be
    • Lachlan's avatar
      Add a pipeline cache for all web pages created by jobs in a given pipeline · 3b8165da
      Lachlan authored
      * Clear the pipeline cache at the start of each new pipeline run.
      
      * Pipeline cache will be part of the artifact.zip and used to generate
        the html pages in the gitlab.io instance.
      3b8165da
    • Lachlan's avatar
      Add nominal script for creating html pages · af1544f2
      Lachlan authored
      - New file publishresults.py
      
        * Takes a path for a set of graph results.
        * Takes an output path for the generated html pages
        * Takes a path for the template files that should
          be used as a base for generating the html files.
      
        * Iterates through through all .png graph files in source path and generates
          a basic html page by substituting stub references in a copy of the
          relevant template.
      
          Copies relevant .png file to specific directory for referencing by
          generated web page.
      
        * Creates an index file that references each generated web page, by
          appending new index lines in a copy of the index template.
      af1544f2
    • Lachlan's avatar
      Add nominal index page · 14094b2b
      Lachlan authored
      * Add html template for index pages which will direct people
      * to other html pages that contain graph results.
      14094b2b
    • Lachlan's avatar
      Add template of nominal webpage · fedc3a46
      Lachlan authored
      * This represents a basic template for the web page that will be
        displayed for each graph.
      * Stubs are used to represent values that will be changed for each
        graph file that will be displayed.
      fedc3a46
    • Lachlan's avatar
      Add copyright notices to files · da8527c4
      Lachlan authored
      da8527c4
    • Lachlan's avatar
      Add trigger for gitlab.io instance · 15dbb1a7
      Lachlan authored
      * Triggers on Master branch only.
      * Does curl request in form requested by gitlab ci.
      * Artifact include all output paths and caches.
      15dbb1a7
  2. 29 Nov, 2018 1 commit
    • Lachlan's avatar
      Change runner to use CI script and cache results · 5d00377c
      Lachlan authored
      * Now use CI script to run benchmark based upon required logic.
      
      * Results are cached (in results_cache) and presented as part of the artifact for
        each job.
      
      * Results for each individual job are stored in results_out, these
        are not cached across jobs.
      5d00377c
  3. 23 Nov, 2018 4 commits
    • Lachlan's avatar
      Add CI Script · bcc0073c
      Lachlan authored
      * .gitlab-ci.yml
        - Add references for CI branch and job name that triggered build,
          allows explicit check for master/development cases.
      
      * New file
        - ci_run.sh
          - Script added which will drive the ci process explicitly.
          - Checks which branch is being considered.
             - For master the results are cached and the overall result
               generated from the cache path only.
             - For non master the results are used as interim comparables
               only and the generated results are not cached.
          - Creates output paths
          - For master, copies the web pages to the pipeline for later
            use in the gitlab.io instance.
      bcc0073c
    • Lachlan's avatar
    • Lachlan's avatar
      ddc7ab41
    • Lachlan's avatar
      Add script to get, correlate and plot json results · 6b97c2d3
      Lachlan authored
      New file:
        - graphresults.py
          - Takes a directory path for the historical json results to be processed
            (delineated by benchmark test).
          - Optionally takes a non formal interim result for non master merged
            tests.
          - Takes an output path for graphs that will be generated.
          - Iterates the directory paths for json files only.
          - Iterates json file list and creates dictionaries for each data set
            based on test name and version , parses total-time and max-rss-kb and
            creates average.
          - Creates a plot for each data grouping (test name and version) and
            deposits in output directory.
      6b97c2d3
  4. 20 Nov, 2018 1 commit
    • Lachlan's avatar
      Add output directories · 27776aac
      Lachlan authored
      * Add results_cache directory
        - caches all json results for multiple pipeline runs.
      * Add pipeline cache directory
        - caches all the graphing and html results for a given pipeline run.
        - Public directory added to reflect gitlab.io structure.
      * Add results_out directory
        - Results for a given job, both json, graph and web page.
      27776aac
  5. 19 Nov, 2018 1 commit
  6. 05 Nov, 2018 1 commit
    • Lachlan's avatar
      Change yaml to PyYaml · 21ada5ce
      Lachlan authored
      * Currently just yaml is referenced under requirements, this does not
        resolve if an install is attempted. This patch changes it to PyYaml.
      21ada5ce
  7. 09 Oct, 2018 1 commit
  8. 14 Aug, 2018 1 commit
  9. 16 Apr, 2018 1 commit
  10. 13 Apr, 2018 2 commits
  11. 04 Apr, 2018 1 commit
  12. 29 Mar, 2018 3 commits
  13. 23 Mar, 2018 1 commit
  14. 12 Mar, 2018 2 commits