Closed
Milestone
Jun 19, 2022–Aug 1, 2022
Gas Effort
Milestone Gas
Intro
Goals
Make gas benchmark run for every MR and that we provide actionable developer feedback immediately and "catch errors".
- Gas benchmarks must run faster.
- Output of Snoop tool must be usable and actionable for software developers
- Keep snoop is robust.
Subgoals:
- Make benchmark run in "4 hours" while output results still being usable.
1. Evaluate potential machines to run benchmark on (cloud / non-cloud) (parallel / single core).
- Build tool/method for evaluating variance of repeated or parallel runs.
- Parallel vs. single core.
- New instances in cloud vs. same instance (I.e. cloud vs fixed hardware).
- Include data generation (sapling, michelson)
- Make snoop useful within a CI run with that output is actionable by software developers such that they
- Short developer feedback loop and be able to run benchmarks locally/remote (ie. without waiting for without waiting for CI)
- Make benchmark able run for each MR (when requested).
TODO: Add more / break down.
Docs
- Design / planning: https://hackmd.io/nuC4R4p0SDe8ovcRZhx5xw
- Meeting notes: https://hackmd.io/kaD-o_RPS7OSxfOA_Ch86Q
Risks
- Dependency on crew externals reviews within the crew.
- GAS maintenance / requests prioritization vs. this crew.
Plan
The plan will be revised every 3-4 week, the upcoming week is described in the most detail, the 2-3 month in high level targets.
June + July
Ongoing: Build tool/method for evaluating variance of repeated or parallel runs.
-
(first attempt) Clearly specify the problem we are trying to solve and define the the metric that is good enough for purposes of gas -
Aws credentials + access. -
Store full data in workloads (#3265 - closed) -
(Don’t crash) Store full data in workloads (#3265 - closed) -
Refactor the signature of the benchmarks !xxxx -
Update gas for K (#3309 - closed) -
Structural namespaces for benchmarks (#3253 - closed) (almost done) -
Example by name, tag or model: ./tool --include-name glob --include-tags glob-a.
-
-
Implement above method. (July). -
Add tezos-snoop as part of the docker image for... (#3351 - closed) - Ec2
- Bare Metal
-
Automate benchmark run (with parameters ) - What benchmarks to run
- How many times to run
- Paralle
-
Persist results on s3. -
Tool to compare multiple results against (Too big for july?) Extra: -
Measure on multi core machines.
-
Out of scope: - Data generation for sapling/michelson. - Running on multiple machines/gathering results.
Demo goal:
- Explain measurement metric.
- How to run and persists results on s3
- Show results for evaluating
- Repeated runs on same instance
- Repeated runs on new instances (destroy/create)
- Optional: Repeated runs on multiple cores.
Risks: - Cloud part / lack of experience.