Proposal: run tooling specs in a separate job
Proposal
Have a dedicated tooling spec
job in pipelines. This job will run most of the specs under spec/scripts
and spec/dot_gitlab_ci
.
Alternative suggestion
I have considered running it in .rspec-fail-fast
initially, but this job also doesn't run in all MRs, for example Support .index.yml config for ruby-lsp (!146069 - merged). We can alternatively change the rule for rspec-fail-fast
so it runs in all MRs, and have this job run the rules_spec.
Context
- While working on Fix spec/dot_gitlab_ci/rules_spec.rb (!146275 - merged), I realize that in order to prevent this spec from failing again, we should ideally run
spec/dot_gitlab_ci/rules_spec.rb
in ALL MRs, because the spec evaluates every file in the project and any file addition/removal could potentially break this test, making it very fragile. - I have learned that tooling specs are running as a part of
rspec unit pg14
with 32 parallelized jobs. We should avoid adding another 32 parallelized jobs so makingrspec unit pg14
run in all MRs is not an option. - Tooling specs shouldn't need to run in all of the pg permutation jobs anyway, so with this change, we may be able to reduce the number of tests that we run in pipeline critical path by extracting tooling specs out of the pg unit jobs.
Technical details
the tooling specs can be identified through the tooling
feature category. We can also have a separate tag to identify them. With this tag, we can skip them in the rspec unit pg14
jobs.
Other thoughts
while we are discussing spec/dot_gitlab_ci/rules.spec
, I believe we should also skip .gitignore
files, because right now this test cannot pass for me locally at all. This is because I have many files that are "gitignored" and they do not match any of our file patterns.