Document how the Chaos scenario is set up

José M. Requena Plensrequested to merge
jmrp/gitlab-qa:jmrp-gitlab-qa-chaos-docs into master

What does this MR do and why?

Closes #691 (closed)

The Chaos section of docs/what_tests_can_be_run.md did not explain how the proxy fits in, which is what the review of gitlab!98019 (closed) asked for, and it had three mistakes:

  • It named the scenario Test::Instance::Chaos. That class does not exist, so the documented command fails with uninitialized constant Gitlab::QA::Scenario::Test::Instance::Chaos. The scenario is Test::Integration::Chaos.
  • Its example passed no release, while perform requires one (CE, EE or an image address).
  • It said the scenario runs only the tests tagged :chaos. The scenario inherits @tag = nil from Test::Integration::Praefect, so it runs Test::Instance::All with no :chaos filter unless you pass -- --tag chaos, which is what the ee:chaos CI job did until it was removed in 30d11584 (March 2023).

I rewrote the section from the code:

  • How the scenario is set up: the chaos Toxiproxy container and its API port, the POST /populate request and why it is sent from a container on the network, a table of the five proxies with where each one forwards, the chaos.test addresses and ports the scenario sets, and what is meant to run after that.
  • Writing a chaos test: the existing sample, plus what toxic and apply do and where to point the Toxiproxy client from the host and from the GitLab QA container.
  • Current status: while checking each step against master (d18bf530) I found that the scenario no longer runs, so the page now says so.

That last part is where I would like a maintainer's view:

  1. Test::Integration::Chaos#perform calls run_gitaly_cluster, which the scenario lost when Gitaly and Praefect moved into components (f66dd679, December 2022). It now raises NoMethodError right after it starts the Toxiproxy container, and removes that container again.
  2. The addresses it sets (@praefect_addr = "chaos.test" and the rest) are no longer read. GitLab, Praefect and Gitaly take theirs from Component::GitalyCluster::GitalyClusterConfig, which derives them from the node names and keeps Gitaly on port 8075, while the three Gitaly proxies forward to 8076, 8077 and 8078.
  3. The QA suite in gitlab-org/gitlab has no tests tagged :chaos and no Toxiproxy dependency. gitlab!98019 (closed), which proposed the first tests, was closed without being merged.

I kept this MR to documentation, as the issue asks, and wrote down the state as it is so that nobody follows the example and wonders why it fails. If you would rather fix the scenario or remove it, I am happy to do either in a separate MR, but I did not want to make that call in a docs change.

!1464 (merged) edits other sections of the same file and says it left the Chaos section to this issue. The two branches merge without conflicts (git merge-tree against its head 8c60d365).

Screenshots or screen recordings

Not applicable, documentation only. The runs behind every statement in the new section are below.

The documented command, and the scenario under its real name (Ruby 3.3.7, master at d18bf530)

gitlab-qa Test::Instance::Chaos:

/src/lib/gitlab/qa/runner.rb:93:in `run': uninitialized constant Gitlab::QA::Scenario::Test::Instance::Chaos (NameError)

gitlab-qa Test::Integration::Chaos EE, with QA_DOCKER_NETWORK=w6chaos so that the run did not create a test network on my machine:

INFO  -- Shell command: `docker run -d  --name chaos --network w6chaos --publish 8474:8474 ghcr.io/shopify/toxiproxy:2.5.0`
INFO  -- Shell command: `docker run --network w6chaos --rm curlimages/curl:7.85.0   -i -s   -X POST http://chaos:8474/populate   -H 'Content-Type: application/json'   -d '[{"name":"postgres","listen":"[::]:5432","upstream":"postgres.test:5432","enabled":true},{"name":"praefect","listen":"[::]:2305","upstream":"praefect.test:2305","enabled":true},{"name":"gitaly1","listen":"[::]:8076","upstream":"gitaly1.test:8076","enabled":true},{"name":"gitaly2","listen":"[::]:8077","upstream":"gitaly2.test:8077","enabled":true},{"name":"gitaly3","listen":"[::]:8078","upstream":"gitaly3.test:8078","enabled":true}]'
INFO  -- Shell command: `docker ps -f name=chaos`
INFO  -- Shell command: `docker rm -f chaos`
/src/lib/gitlab/qa/scenario/test/integration/chaos.rb:25:in `perform': undefined method `run_gitaly_cluster' for an instance of Gitlab::QA::Scenario::Test::Integration::Chaos (NoMethodError)

The upstream hosts stay on .test although the network was w6chaos.

What the scenario object holds, and what the containers would be configured with 
perform arity: [[:req, :release], [:rest, :rspec_args]]
scenario responds to run_gitaly_cluster: false
component responds to run_gitaly_cluster: true
@tag: nil; @spec_suite: "Test::Instance::All"; @env: {:QA_PRAEFECT_REPOSITORY_STORAGE=>"default"}
config used for omnibus: praefect_addr=praefect.test praefect_port=2305 database=postgres.test:5432 primary=gitaly1.test:8075 secondary=gitaly2.test:8075 tertiary=gitaly3.test:8075
ivars set by Chaos#initialize: @praefect_addr="chaos.test" @database_addr="chaos.test" @primary_node_addr="chaos.test" @primary_node_port=8076
omnibus gitaly_address line: 'gitaly_address' => "tcp://praefect.test:2305",
component config: praefect=praefect.test:2305 database=postgres.test:5432 gitaly ports=[8075, 8075, 8075]
praefect omnibus: host: 'postgres.test', | 'address': 'tcp://gitaly1.test:8075', | 'address': 'tcp://gitaly2.test:8075', | 'address': 'tcp://gitaly3.test:8075',
gitaly default port: 8075

The first config is the scenario's own, which writes the GitLab Omnibus configuration; the second is the one Component::GitalyCluster uses for Praefect and the Gitaly nodes.

The proxies and the Toxiproxy client example, against the container the component starts

I started only Component::Chaos (Gitlab::QA::Component::Chaos.new.start) and left it running.

From the host, through the published port:

$ curl -s localhost:8474/proxies | jq -c 'to_entries[] | {name: .key, listen: .value.listen, upstream: .value.upstream}'
{"name":"gitaly1","listen":"[::]:8076","upstream":"gitaly1.test:8076"}
{"name":"gitaly2","listen":"[::]:8077","upstream":"gitaly2.test:8077"}
{"name":"gitaly3","listen":"[::]:8078","upstream":"gitaly3.test:8078"}
{"name":"postgres","listen":"[::]:5432","upstream":"postgres.test:5432"}
{"name":"praefect","listen":"[::]:2305","upstream":"praefect.test:2305"}

From a container on the same network, with the toxiproxy gem (2.0.2), Toxiproxy.host = 'http://chaos.w6chaos:8474' and the toxic from the sample:

toxics on gitaly1 before: []
toxics on gitaly1 inside apply: [["latency", "downstream", {"latency"=>30000, "jitter"=>0}]]
toxics on gitaly1 after: []

With no host set, the client used http://127.0.0.1:8474 and listed the same five proxies from the host. A container on a user-defined network resolved <container>.<network> (here chaos.w6chaos), which is what chaos.test relies on.

The POST /populate in the scenario run reached the container by name (http://chaos:8474), which is the address the page now gives the client. I ran the same client script with Toxiproxy.host = 'http://chaos:8474' from a container on another throwaway network, w6chaos2, and checked which names resolve there:

client 2.0.2, uri http://chaos:8474
toxics on gitaly1 before: []
toxics on gitaly1 inside apply: [["latency", "downstream", {"latency"=>30000, "jitter"=>0}]]
toxics on gitaly1 after: []
chaos: resolves
chaos.w6chaos2: resolves
chaos.test: does not resolve

chaos.test resolves only when the network is named test, so the page uses the container name. In the client source (2.0.2), toxic is alias_method :toxic, :downstream, and ToxicCollection#apply destroys the toxics in an ensure.

How to set up and validate locally

  1. Check out this branch and run bundle install.
  2. Run the old command, bundle exec exe/gitlab-qa Test::Instance::Chaos, and see the NameError.
  3. Run bundle exec exe/gitlab-qa Test::Integration::Chaos EE. It pulls ghcr.io/shopify/toxiproxy:2.5.0 and curlimages/curl:7.85.0, creates the test Docker network if it is missing (the network stays afterwards), starts the chaos container with port 8474 published, populates it, removes it and stops with the NoMethodError above.
  4. To check the proxy table and the client example, start only the component with bundle exec ruby -Ilib -e "require 'gitlab/qa'; Gitlab::QA::Component::Chaos.new.start", then run curl -s localhost:8474/proxies. Remove it afterwards with docker rm -f chaos.
  5. To see the addresses the containers would get, run bundle exec ruby -Ilib -e "require 'gitlab/qa'; c = Gitlab::QA::Scenario::Test::Integration::Chaos.new.config; p [c.praefect_addr, c.database_node_addr, c.primary_node_addr, c.primary_node_port]", which prints ["praefect.test", "postgres.test", "gitaly1.test", 8075].

I also ran the project's CI checks on the branch with Ruby 3.3.7: bundle exec rubocop (154 files, no offenses) and bundle exec rspec (446 examples, 0 failures). Neither reads the docs, but both pass.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports