rspec fast_spec_helper, rspec unit clickhouse and rspec-ee unit clickhouse jobs failed to create
Problems
We are seeing invalid yaml syntax error in this MR pipeline: !133322 (merged).
Unable to create pipeline
'rspec fast_spec_helper' job needs 'setup-test-env' job, but 'setup-test-env' is not in any previous stage
'rspec fast_spec_helper' job needs 'retrieve-tests-metadata' job, but 'retrieve-tests-metadata' is not in any previous stage
'rspec fast_spec_helper' job needs 'compile-test-assets' job, but 'compile-test-assets' is not in any previous stage
'rspec unit clickhouse' job needs 'setup-test-env' job, but 'setup-test-env' is not in any previous stage
'rspec unit clickhouse' job needs 'retrieve-tests-metadata' job, but 'retrieve-tests-metadata' is not in any previous stage
'rspec unit clickhouse' job needs 'compile-test-assets' job, but 'compile-test-assets' is not in any previous stage
'rspec-ee unit clickhouse' job needs 'setup-test-env' job, but 'setup-test-env' is not in any previous stage
'rspec-ee unit clickhouse' job needs 'retrieve-tests-metadata' job, but 'retrieve-tests-metadata' is not in any previous stage
'rspec-ee unit clickhouse' job needs 'compile-test-assets' job, but 'compile-test-assets' is not in any previous stage
Initial invesgation resulted in 2 questions:
- why didn't this MR run merged results pipeline? In this earlier MR, it was able to run merged results pipeline, but it didn't for the pipeline we are looking at now. UPDATE: After reaching out to pipeline execution team, I now think this is due to the yaml syntax error preventing merged result pipeline from creating.
- Which rule is missing that causes those setup jobs to not be created?
Culprit
mismatched job rules likely introduced by 239f0bd0. We see the following rule mismatch:
rspec fast_spec_helper
is triggered when the following patterns are changed:
.rails:rules:ee-and-foss-fast_spec_helper:
rules:
- <<: *if-merge-request-labels-run-all-rspec
- <<: *if-merge-request
changes: *backend-patterns. <---- look here
- <<: *if-default-refs
changes: *core-backend-patterns
rspec unit clickhouse
and rspec-ee unit clickhouse
are triggered when the following patterns are changed:
.rails:rules:clickhouse-changes:
rules:
- <<: *if-merge-request
changes: *backend-patterns <---- look here
and backend-patterns
contains gems/**/*
now:
- "{,jh/}Gemfile{,.lock}"
- "Rakefile"
- "config.ru"
# List explicitly all the app/ dirs that are backend (i.e. all except app/assets).
- "{,ee/,jh/}{app/channels,app/components,app/controllers,app/finders,app/graphql,app/helpers,app/mailers,app/models,app/policies,app/presenters,app/serializers,app/services,app/uploaders,app/validators,app/views,app/workers}/**/*"
- "{,ee/,jh/}{bin,config,db,generator_templates,lib,gems}/**/*". <---- `gems/**/*` is now a part of the backend patterns
- "{,ee/,jh/}spec/**/*"
# CI changes
- ".gitlab-ci.yml"
- ".gitlab/ci/**/*"
- "*_VERSION"
- "scripts/rspec_helpers.sh"
# Mapped patterns (see tests.yml)
- "data/whats_new/*.yml"
- "doc/index.md"
However, setup-test-env
is only triggered in MRs pipelines when labeled with pipeline:run-all-rspec:
.rails:rules:setup-test-env:
rules:
- <<: *if-default-refs
changes: *setup-test-env-patterns
- <<: *if-merge-request-labels-run-all-rspec
Similarly, retrieve-tests-metadata
runs when MR is labeled with pipeline:run-all-rspec, or frontend + feature flag .
.test-metadata:rules:retrieve-tests-metadata:
rules:
- <<: *if-default-refs
changes: *code-backstage-patterns
- <<: *if-default-refs
changes: *workhorse-patterns
- <<: *if-merge-request-labels-run-all-rspec
- <<: *if-merge-request-labels-frontend-and-feature-flag
Finally, compile-test-assets
is triggered when MR is labeled with pipeline:run-all-rspec .
.frontend:rules:compile-test-assets:
rules:
- <<: *if-merge-request-labels-run-all-rspec
- <<: *if-default-refs
changes: *code-backstage-qa-patterns
- <<: *if-default-refs
changes: *workhorse-patterns
so fast_spec_helper
, rspec unit clickhouse
and rspec-ee unit clickhouse
jobs are created, while setup-test-env
, retrieve-tests-metadata
, and compile-test-assets
aren't.
Proposed solutions
- add
- <<: *if-merge-request
changes: *backend-patterns.
to the following rules:
.rails:rules:setup-test-env
.frontend:rules:compile-test-assets
.test-metadata:rules:retrieve-tests-metadata
However, this may result in these 3 prep jobs running more than we need? But knowing that we do want to always run rspec tests when changes are made to gems
, without creating a separate pattern category for gems, this is the only option that will solve our problem right now.
-
Alternatively, we can revert !132876 (merged), and we figure out a better pattern category for
gems/**/*
. Perhaps puttinggems
underbackend-patterns
was not ideal, and they may deserve their own pattern category? -
This is more of a long term improvement - Do we really need
compile-test-assets
to unit testclickhouse
which is a gem? I realize it's inheriting dependencies fromrspec-base
. But it feels like an overkill to require frontend assets to test a gem change? But to resolve this issue, we will have to refactor the configuration for clickhouse and fast spec helper so they don't inherit this dependency fromrspec-base
. I don't know if this is possible without spending some time learning how these jobs are setup.
Tagging @gl-quality/eng-prod because I realize this is a generalized problem for all MRs making changes to the gems
folder now. We should reconsider what tests are necessary to run for gems
changes, whether gems
belong to backend-patterns
or should they deserve a new pattern type, etc. In the mean time, to quickly unblock affected MRs, I have opened Run rspec prep jobs for backend changes (!133381 - merged)