Improve Knapsack cache
It seems that our Knapsack cache is not efficient as described here:
- https://gitlab.com/gitlab-com/infrastructure/issues/1057
- https://gitlab.com/gitlab-org/gitlab-ce/issues/29623
There are two options:
- Use S3 to store knapsack cache,
- Commit knapsack results to repo.
1. Instead of relying on an internal cache we should store test results on created S3 bucket and download that as part of the CI job:
knapsack:
<<: *knapsack-state
<<: *dedicated-runner
stage: prepare
script:
- mkdir -p knapsack/
- wget -O knapsack/rspec_report.json https://s3-bucket.knapsack-cache.com/knapsack/rspec_report.json
- wget -O knapsack/spinach_report.json https://s3-bucket.knapsack-cache.com/knapsack/spinach_report.json
- '[[ -f knapsack/rspec_report.json ]] || echo "{}" > knapsack/rspec_report.json'
- '[[ -f knapsack/spinach_report.json ]] || echo "{}" > knapsack/spinach_report.json'
update-knapsack:
<<: *knapsack-state
<<: *dedicated-runner
stage: post-test
script:
- scripts/merge-reports knapsack/rspec_report.json knapsack/rspec_node_*.json
- scripts/merge-reports knapsack/spinach_report.json knapsack/spinach_node_*.json
- s3 put knapsack/rspec_report.json s3-bucket.knapsack-cache.com/knapsack/rspec_report.json
- s3 put knapsack/spinach_report.json s3-bucket.knapsack-cache.com/knapsack/spinach_report.json
only:
- master@gitlab-org/gitlab-ce
- master@gitlab-org/gitlab-ee
- master@gitlab/gitlabhq
- master@gitlab/gitlab-ee
2. Second way would be to commit knapsack results with repository and update that every sometime, like one release manually.
cc @rymai