Proposed new tool to replace run.sh

Motivation

Following issue #103 and %Redesign Test Harness, I have written a quick proof of concept (the code needs some cleanup) to show what an alternative to run.sh should do.

Implementation

Tool Functionality

• more easily specify version combinations
• flexible manifest format for defining versions to build
• generate docker-compose files for arbitrary combinations of client and server
• arbitrarily define the versions of client / server to build
• plug various config files per version
• plug various build backends per version
• run 1 or more tests locally using docker-compose
• run tests with a configurable degree of parallelism
• fancy pass/fail view for local use
• ability to export run data (pass/fail, exit code, stdout, stderr) as json
• render out docker compose files for a given combinations of client / server
• render out a gitlab ci file for given combinations of client / server
• building + running arbitrary versions of client / server
• parallel building of client / server images
  • allow defining custom names for services (to prevent clashes)
• pushing built images to a registry
• attempting to pull images from a registry before building
  • only build images that weren't pulled
• isolate various jobs from one-another
  • create a docker network for each combination (this is done automatically)
  • run each job (and keep each set of volumes) is a different folder

CI Functionality

• support asset hub
• support pants
• generate ci pipeline from manifest
• have CI commit results to a central jobs.json file
• have CI pass the jobs in jobs.json in as exclude jobs to the cli

Race Conditions

As a side effect of generating the entire run definition at once, we are now affected by race conditions in most of the tests:

• buildgrid worker crashes when it cannot reach the runner
• recc crashes when it cannot reach the server

Because we are listening for the exit code of the client service with the --exit-code-from flag, docker-compose automatically applies the --always-recreate-deps flag. Any crashes whatsoever will bring down the entire system.

Ideally this means we can maintain backwards compatibility with old versions of tools and instead publish a different builder / config combination for new versions as needed. By exporting run information to well-formatted json, we can maintain a backlog of old runs as well such that even if bazel@1.2.1+buildfarm@0.3.0 (versions made up) is no longer covered by the harness, we still have data for it.

Additionally, it will make the local testing use-case much more compelling as you can run multiple builds at once.

Outcomes

Closes #88 (closed), and Closes #89, Closes #92, Closes #101 (closed), Closes #103, Closes #104, Closes #109, Closes #106