Remove the Chaos scenario

José M. Requena Plensrequested to merge
jmrp/gitlab-qa:jmrp-remove-chaos-scenario into master

What does this MR do and why?

@mlapierre, as you suggested in !1465 (closed), this removes the Chaos scenario and its documentation.

Related to #691 (closed) and !1465 (closed).

When I documented the scenario in !1465 (closed), I noted that the QA suite in gitlab-org/gitlab has no tests tagged :chaos. In your note there you confirmed it, added that there has been no follow-up work in the years since the scenario was introduced and that the chaos testing now proposed for Gitaly will not use the QA test framework, and closed both the issue and that MR, since updating the docs would suggest QA chaos tests are still planned. You asked for the scenario code and documentation to be removed in a separate MR instead. This is that MR.

What it removes:

  • lib/gitlab/qa/scenario/test/integration/chaos.rb: the Test::Integration::Chaos scenario.
  • lib/gitlab/qa/component/chaos.rb: Component::Chaos, the Toxiproxy container the scenario started. Only the scenario used it.
  • The Chaos section of docs/what_tests_can_be_run.md, which named the scenario Test::Instance::Chaos, a class that never existed.

What it keeps: Test::Integration::Praefect, which the Chaos scenario inherited from and which the praefect and gitaly-reftables-backend jobs of .gitlab/ci/test-on-omnibus/main.gitlab-ci.yml in gitlab-org/gitlab still use, and Component::GitalyCluster with its address and port settings, which the Gitaly Cluster, Praefect, Airgapped and Repository Storage scenarios use. No spec, CI job, rake task, .rubocop_todo.yml entry or other page referred to the scenario, so nothing else changes. After this branch, git grep -i -E 'chaos|toxi' finds nothing.

The scenario had not worked since the Gitaly and Praefect setup moved into components in f66dd679: perform still called run_gitaly_cluster, which the scenario no longer has, so it raised NoMethodError right after starting the Toxiproxy container, and the proxy addresses it set were no longer read by anything.

The project builds its changelog from commit trailers, so the commit carries Changelog: removed, as the template asks.

The branch merges with !1464, which edits other sections of the same page, without conflicts.

Screenshots or screen recordings

Not applicable, this removes code and documentation. The checks I ran are below.

rubocop and rspec on master and on this branch (Ruby 3.3.7, Bundler 2.4.5, as the project's CI runs them)

master at 702b1227:

154 files inspected, no offenses detected
446 examples, 0 failures

This branch:

152 files inspected, no offenses detected
446 examples, 0 failures

The two files fewer are the two removed files. No spec covered the scenario, so the example count does not change.

What the scenario did on master

With Component::Chaos replaced by an object that starts nothing, so that no container runs:

scenario responds to run_gitaly_cluster: false
addresses the scenario sets: {:@praefect_addr=>"chaos.test", :@database_addr=>"chaos.test", :@primary_node_addr=>"chaos.test", :@primary_node_port=>8076, :@secondary_node_addr=>"chaos.test", :@secondary_node_port=>8077, :@tertiary_node_addr=>"chaos.test", :@tertiary_node_port=>8078}
addresses the configuration hands GitLab: praefect=praefect.test:2305 gitaly=gitaly1.test,gitaly2.test,gitaly3.test ports=[8075, 8075, 8075]
perform raises NoMethodError: undefined method `run_gitaly_cluster' for an instance of Gitlab::QA::Scenario::Test::Integration::Chaos
Loading every file under lib/gitlab/qa, on master and on this branch

Every file is loaded through the gem's Zeitwerk loader, one at a time and in path order, in a single process, so that one failure does not hide the rest. Two files fail to load that way on master too, and are not touched here: component/gitaly.rb extends Forwardable and only loads once something has required forwardable, which component/gitlab.rb and runtime/logger.rb do but which come after it in path order, and scenario/test/integration/client_ssl.rb defines ClientSSL where the loader expects ClientSsl.

master:

files under lib/gitlab/qa: 114, loaded: 112
not loaded (2):
  component/gitaly.rb: NameError
  scenario/test/integration/client_ssl.rb: Zeitwerk::NameError
Scenario::Test::Integration::Chaos defined: true
Component::Chaos defined: true
subclasses of the Praefect scenario: ["Gitlab::QA::Scenario::Test::Integration::Chaos"]

This branch:

files under lib/gitlab/qa: 112, loaded: 110
not loaded (2):
  component/gitaly.rb: NameError
  scenario/test/integration/client_ssl.rb: Zeitwerk::NameError
Scenario::Test::Integration::Chaos defined: false
Component::Chaos defined: false
subclasses of the Praefect scenario: []

bundle exec exe/gitlab-qa Test::Integration::Chaos EE on this branch now stops before starting any container:

/src/lib/gitlab/qa/runner.rb:93:in `run': uninitialized constant Gitlab::QA::Scenario::Test::Integration::Chaos (NameError)
markdownlint on docs/what_tests_can_be_run.md

The project runs no docs linter, so I ran markdownlint-cli2 with its defaults on the page from master and from this branch. It reports 398 findings on master and 394 here. The four that go are in the removed section (three line lengths and one $ before a command), and none is added.

How to set up and validate locally

  1. Check out this branch and run bundle install.
  2. Run bundle exec rubocop and bundle exec rspec.
  3. Run git grep -i -E 'chaos|toxi', which finds nothing.
  4. Run bundle exec exe/gitlab-qa Test::Integration::Chaos EE, which stops with uninitialized constant Gitlab::QA::Scenario::Test::Integration::Chaos before it starts any container.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports