gitaly-cluster

Passed Started 1 year ago by
1Running with gitlab-runner 16.3.1 (f5dfa4d1)
2  on gitlab-org-qa-runner ss7o3c7p, system ID: s_6a7379124369
 3  Resolving secrets  
  00:00
 4  Preparing the "docker+machine" executor  
  01:14
5Using Docker executor with image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 ...
6Starting service docker:24.0.5-dind ...
7Pulling docker image docker:24.0.5-dind ...
8Using docker image sha256:7015f2c475d511a251955877c2862016a4042512ba625ed905e69202f87e1a21 for docker:24.0.5-dind with digest docker@sha256:3c6e4dca7a63c9a32a4e00da40461ce067f255987ccc9721cf18ffa087bcd1ef ...
9Waiting for services to be up and running (timeout 30 seconds)...
10Authenticating with credentials from job payload (GitLab Registry)
11Pulling docker image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 ...
12Using docker image sha256:5b6b9b1e9933d18667b8e9caf69bab048855b4e7bd307ec442f5c62ace584d2a for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:27ab2d3fd889b0e37425d095c34302293ce43da8c6fbe5876f07327517f9859c ...
 13  Preparing environment  
  00:00
14Running on runner-ss7o3c7p-project-278964-concurrent-0 via runner-ss7o3c7p-gitlab-org-qa-1700105055-c7a5936d-gcp...
 15  Getting source from Git repository  
  00:20
16Fetching changes with git depth set to 20...
17Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/
18Created fresh repository.
19remote: Enumerating objects: 155914, done.        
20remote: Counting objects: 100% (155914/155914), done.        
21remote: Compressing objects: 100% (109217/109217), done.        
22remote: Total 155914 (delta 69750), reused 99194 (delta 40527), pack-reused 0        
23Receiving objects: 100% (155914/155914), 137.81 MiB | 44.04 MiB/s, done.
24Resolving deltas: 100% (69750/69750), done.
25From https://gitlab.com/gitlab-org/gitlab
26 * [new ref]             refs/pipelines/1074263059 -> refs/pipelines/1074263059
27 * [new branch]          master                    -> origin/master
28Checking out 7f3e0a47 as detached HEAD (ref is master)...
29Skipping Git submodules setup
 30  Restoring cache  
  00:05
31Checking cache for qa-e2e-ruby-3.0-5fe14f76d4eb79aabf540b0a58e6de08-16...
32Downloading cache from https://storage.googleapis.com/gitlab-qa-runner-cache-1/project/278964/qa-e2e-ruby-3.0-5fe14f76d4eb79aabf540b0a58e6de08-16 
33WARNING: .gems/gems/deprecation_toolkit-2.0.3/gemfiles/test/deprecations: chmod .gems/gems/deprecation_toolkit-2.0.3/gemfiles/test/deprecations: no such file or directory (suppressing repeats) 
34Successfully extracted cache
 35  Downloading artifacts  
  00:01
36Downloading artifacts for download-knapsack-report (5547762632)...
37Downloading artifacts from coordinator... ok        host=storage.googleapis.com id=5547762632 responseStatus=200 OK token=64_AaMvy
38Downloading artifacts for download-fast-quarantine-report (5547762633)...
39Downloading artifacts from coordinator... ok        host=storage.googleapis.com id=5547762633 responseStatus=200 OK token=64_AaMvy
 40  Executing "step_script" stage of the job script  
  15:46
41Using docker image sha256:5b6b9b1e9933d18667b8e9caf69bab048855b4e7bd307ec442f5c62ace584d2a for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:27ab2d3fd889b0e37425d095c34302293ce43da8c6fbe5876f07327517f9859c ...
42$ echo "SUITE_RAN=true" > $QA_SUITE_STATUS_ENV_FILE
43$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
44$ if [ "$RUN_WITH_BUNDLE" == "true" ]; then # collapsed multi-line command
45Bundle complete! 38 Gemfile dependencies, 136 gems now installed.
46Use `bundle info [gemname]` to see where a bundled gem is installed.
47$ export RSPEC_FAST_QUARANTINE_PATH="/home/gitlab/qa/${RSPEC_FAST_QUARANTINE_PATH}"
48$ export RSPEC_SKIPPED_TESTS_REPORT_PATH="/home/gitlab/qa/rspec/skipped_tests-${CI_JOB_ID}.txt"
49$ export QA_COMMAND="$BUNDLE_PREFIX gitlab-qa ${QA_SCENARIO:=Test::Instance::Image} $RELEASE $GITLAB_QA_OPTS -- $QA_TESTS $QA_RSPEC_TAGS $RSPEC_REPORT_OPTS"
50$ echo "Running - '$QA_COMMAND'"
51Running - 'bundle exec gitlab-qa Test::Integration::GitalyCluster registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255  --   --format QA::Support::JsonFormatter --out tmp/rspec-5547762662.json --format RspecJunitFormatter --out tmp/rspec-5547762662.xml --format html --out tmp/rspec-$(echo gitaly-cluster | sed 's/[ /]/_/g').htm --color --format documentation'
52$ eval "$QA_COMMAND"
53[Nov 16 2023 03:31:26 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q postgres:11`
54[Nov 16 2023 03:31:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^postgres$`
55[Nov 16 2023 03:31:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
56[Nov 16 2023 03:31:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker network create test`
57[Nov 16 2023 03:31:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name postgres --net test --env POSTGRES_PASSWORD="SQL_PASSWORD" postgres:11`
58[Nov 16 2023 03:31:33 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`
59[Nov 16 2023 03:31:33 UTC (Gitlab QA)] ERROR -- Shell command output:
60psql: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
61	Is the server running locally and accepting connections on that socket?
62[Nov 16 2023 03:31:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`
63[Nov 16 2023 03:31:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec postgres bash -c "psql -U postgres -d template1 -c \"CREATE DATABASE praefect_production OWNER postgres\""`
64[Nov 16 2023 03:31:39 UTC (Gitlab QA)] INFO  -- Starting Gitaly Cluster
65[Nov 16 2023 03:31:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^alpine-2b2bbade$`
66[Nov 16 2023 03:31:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
67[Nov 16 2023 03:31:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name alpine-2b2bbade --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z alpine/curl:latest tail -f /dev/null`
68[Nov 16 2023 03:31:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
69[Nov 16 2023 03:31:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.5.0/tls_certificates/authority/. alpine-2b2bbade:/etc/gitlab/trusted-certs`
70[Nov 16 2023 03:31:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.5.0/tls_certificates/gitlab/. alpine-2b2bbade:/etc/gitlab/ssl`
71[Nov 16 2023 03:31:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker ps -f name=alpine-2b2bbade`
72[Nov 16 2023 03:31:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker rm -f alpine-2b2bbade`
73[Nov 16 2023 03:31:51 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
74[Nov 16 2023 03:31:51 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
75[Nov 16 2023 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitlab-gitaly-cluster$`
76[Nov 16 2023 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly1$`
77[Nov 16 2023 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
78[Nov 16 2023 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
79[Nov 16 2023 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitaly1 --net test --hostname gitaly1.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/gitaly1/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
80[Nov 16 2023 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitlab-gitaly-cluster --net test --hostname gitlab-gitaly-cluster.test -p 80 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/gitlab-gitaly-cluster/logs:/var/log/gitlab:Z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
81[Nov 16 2023 03:32:42 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "echo \"postgresql['enable'] = false;
82redis['enable'] = false;
83nginx['enable'] = false;
84puma['enable'] = false;
85sidekiq['enable'] = false;
86gitlab_workhorse['enable'] = false;
87gitlab_rails['rake_cache_clear'] = false;
88gitlab_rails['auto_migrate'] = false;
89gitlab_exporter['enable'] = false;
90gitlab_kas['enable'] = false;
91praefect['enable'] = false;
92prometheus['enable'] = true;
93gitaly['enable'] = true;
94gitaly['configuration'] = {
95  'listen_addr': '0.0.0.0:8075',
96  'prometheus_listen_addr': '0.0.0.0:9236',
97  'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
98}
99gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
100gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
101git_data_dirs({
102  'gitaly1' => {
103    'path' => '/var/opt/gitlab/git-data'
104  },
105  'gitaly2' => {
106    'path' => '/var/opt/gitlab/git-data'
107  },
108  'gitaly3' => {
109    'path' => '/var/opt/gitlab/git-data'
110  }
111});
112\" > /etc/gitlab/gitlab.rb;"`
113[Nov 16 2023 03:32:42 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitaly1`
114[Nov 16 2023 03:32:42 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo \"# Generated by GitLab QA Omnibus Configurator at 2023-11-16 03:31:26
115gitlab_rails['gitlab_default_theme'] = 10 # Light Red Theme
116gitlab_rails['gitlab_disable_animations'] = true # Disable animations
117gitlab_rails['application_settings_cache_seconds'] = 0 # Settings cache expiry
118gitlab_rails['initial_root_password'] = '5iveL!fe' # Initial root password
119external_url 'http://gitlab-gitaly-cluster.test';
120git_data_dirs({
121  'default' => {
122    'gitaly_address' => 'tcp://praefect.test:2305',
123    'gitaly_token' => 'PRAEFECT_EXTERNAL_TOKEN'
124  }
125});
126gitaly['enable'] = false;
127gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
128gitlab_rails['env'] = {'GITLAB_LICENSE_MODE'=>'test', 'CUSTOMER_PORTAL_URL'=>'https://customers.staging.gitlab.com', 'GITLAB_QA_USER_AGENT'=>'*****', 'GITLAB_ALLOW_SEPARATE_CI_DATABASE'=>'true'}
129prometheus['scrape_configs'] = [
130  {
131    'job_name' => 'praefect',
132    'static_configs' => [
133      'targets' => [
134        'praefect.test:9652'
135      ]
136    ]
137  },
138  {
139    'job_name' => 'praefect-gitaly',
140    'static_configs' => [
141      'targets' => [
142        'gitaly1.test:9236',
143        'gitaly2.test:9236',
144        'gitaly3.test:9236'
145      ]
146    ]
147  }]\" > /etc/gitlab/gitlab.rb;"`
148[Nov 16 2023 03:32:42 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitlab-gitaly-cluster`
149[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
150[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "echo '#!/usr/bin/env bash
151if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
152  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
153  exit 1
154fi
155' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
156[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
157[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
158[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
159[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly2$`
160[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
161[Nov 16 2023 03:33:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitaly2 --net test --hostname gitaly2.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/gitaly2/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
162[Nov 16 2023 03:33:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "echo \"postgresql['enable'] = false;
163redis['enable'] = false;
164nginx['enable'] = false;
165puma['enable'] = false;
166sidekiq['enable'] = false;
167gitlab_workhorse['enable'] = false;
168gitlab_rails['rake_cache_clear'] = false;
169gitlab_rails['auto_migrate'] = false;
170gitlab_exporter['enable'] = false;
171gitlab_kas['enable'] = false;
172praefect['enable'] = false;
173prometheus['enable'] = true;
174gitaly['enable'] = true;
175gitaly['configuration'] = {
176  'listen_addr': '0.0.0.0:8075',
177  'prometheus_listen_addr': '0.0.0.0:9236',
178  'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
179}
180gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
181gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
182git_data_dirs({
183  'gitaly1' => {
184    'path' => '/var/opt/gitlab/git-data'
185  },
186  'gitaly2' => {
187    'path' => '/var/opt/gitlab/git-data'
188  },
189  'gitaly3' => {
190    'path' => '/var/opt/gitlab/git-data'
191  }
192});
193\" > /etc/gitlab/gitlab.rb;"`
194[Nov 16 2023 03:33:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitaly2`
195[Nov 16 2023 03:34:16 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
196[Nov 16 2023 03:34:16 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "echo '#!/usr/bin/env bash
197if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
198  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
199  exit 1
200fi
201' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
202[Nov 16 2023 03:34:16 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
203[Nov 16 2023 03:34:16 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
204[Nov 16 2023 03:34:16 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
205[Nov 16 2023 03:34:17 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly3$`
206[Nov 16 2023 03:34:17 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
207[Nov 16 2023 03:34:17 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitaly3 --net test --hostname gitaly3.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/gitaly3/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
208[Nov 16 2023 03:34:18 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "echo \"postgresql['enable'] = false;
209redis['enable'] = false;
210nginx['enable'] = false;
211puma['enable'] = false;
212sidekiq['enable'] = false;
213gitlab_workhorse['enable'] = false;
214gitlab_rails['rake_cache_clear'] = false;
215gitlab_rails['auto_migrate'] = false;
216gitlab_exporter['enable'] = false;
217gitlab_kas['enable'] = false;
218praefect['enable'] = false;
219prometheus['enable'] = true;
220gitaly['enable'] = true;
221gitaly['configuration'] = {
222  'listen_addr': '0.0.0.0:8075',
223  'prometheus_listen_addr': '0.0.0.0:9236',
224  'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
225}
226gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
227gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
228git_data_dirs({
229  'gitaly1' => {
230    'path' => '/var/opt/gitlab/git-data'
231  },
232  'gitaly2' => {
233    'path' => '/var/opt/gitlab/git-data'
234  },
235  'gitaly3' => {
236    'path' => '/var/opt/gitlab/git-data'
237  }
238});
239\" > /etc/gitlab/gitlab.rb;"`
240[Nov 16 2023 03:34:18 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitaly3`
241[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
242[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "echo '#!/usr/bin/env bash
243if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
244  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
245  exit 1
246fi
247' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
248[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
249[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
250[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
251[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^praefect$`
252[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
253[Nov 16 2023 03:35:02 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name praefect --net test --hostname praefect.test -p 2305 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/praefect/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
254[Nov 16 2023 03:35:03 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec praefect bash -c "echo \"postgresql['enable'] = false;
255redis['enable'] = false;
256nginx['enable'] = false;
257puma['enable'] = false;
258sidekiq['enable'] = false;
259gitlab_workhorse['enable'] = false;
260gitlab_rails['rake_cache_clear'] = false;
261gitlab_rails['auto_migrate'] = false;
262gitlab_exporter['enable'] = false;
263gitlab_kas['enable'] = false;
264gitaly['enable'] = false;
265prometheus['enable'] = true;
266praefect['enable'] = true;
267praefect['configuration'] = {
268  listen_addr: '0.0.0.0:2305',
269  prometheus_listen_addr: '0.0.0.0:9652',
270  auth: {
271    token: 'PRAEFECT_EXTERNAL_TOKEN'
272  },
273  reconciliation: {
274    scheduling_interval: '10s'
275  },
276  database: {
277    host: 'postgres.test',
278    port: 5432,
279    user: 'postgres',
280    password: 'SQL_PASSWORD',
281    dbname: 'praefect_production',
282    sslmode: 'disable'
283  },
284  virtual_storage: [
285    {
286      name: 'default',
287      node: [
288        {
289          'storage': 'gitaly1',
290          'address': 'tcp://gitaly1.test:8075',
291          'token': 'PRAEFECT_INTERNAL_TOKEN'
292        },
293        {
294          'storage': 'gitaly2',
295          'address': 'tcp://gitaly2.test:8075',
296          'token': 'PRAEFECT_INTERNAL_TOKEN'
297        },
298        {
299          'storage': 'gitaly3',
300          'address': 'tcp://gitaly3.test:8075',
301          'token': 'PRAEFECT_INTERNAL_TOKEN'
302        }
303      ],
304    }
305  ]
306}
307\" > /etc/gitlab/gitlab.rb;"`
308[Nov 16 2023 03:35:03 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false praefect`
309[Nov 16 2023 03:35:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec praefect bash -c "praefect -config /var/opt/gitlab/praefect/cluster_config.toml check || true"`
310[Nov 16 2023 03:35:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' praefect`
311[Nov 16 2023 03:35:47 UTC (Gitlab QA)] INFO  -- Gitaly Cluster Ready
312[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Waiting for GitLab to become healthy ...
313[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-gitaly-cluster`
314[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker port gitlab-gitaly-cluster 80/tcp`
315[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- -> GitLab is available at `http://docker:32769`!
316[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
317[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo '#!/usr/bin/env bash
318if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
319  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
320  exit 1
321fi
322' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
323[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
324[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.5.0/support/data gitlab-gitaly-cluster:/tmp/data-seeds`
325[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Running exec_commands...
326[Nov 16 2023 03:36:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-rails runner /tmp/data-seeds/admin_access_token_seed.rb"`
327[Nov 16 2023 03:36:59 UTC (Gitlab QA)] INFO  -- Running Gitaly Cluster specs!
328[Nov 16 2023 03:36:59 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
329[Nov 16 2023 03:36:59 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255`
330[Nov 16 2023 03:38:00 UTC (Gitlab QA)] INFO  -- Running test suite `Test::Instance::All` for gitlab-ee
331[Nov 16 2023 03:38:00 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -t --rm --net=test --env QA_AWS_S3_REGION="$QA_AWS_S3_REGION" --env QA_FACEBOOK_OAUTH_APP_ID="$QA_FACEBOOK_OAUTH_APP_ID" --env QA_GOOGLE_CDN_JSON_KEY="$QA_GOOGLE_CDN_JSON_KEY" --env QA_SUITE_STATUS_ENV_FILE="$QA_SUITE_STATUS_ENV_FILE" --env QA_THIRD_PARTY_DOCKER_REGISTRY="$QA_THIRD_PARTY_DOCKER_REGISTRY" --env QA_RSPEC_REPORT_PATH="$QA_RSPEC_REPORT_PATH" --env QA_SAVE_TEST_METRICS="$QA_SAVE_TEST_METRICS" --env QA_GOOGLE_CDN_SIGNURL_KEY_NAME="$QA_GOOGLE_CDN_SIGNURL_KEY_NAME" --env QA_1P_PASSWORD="$QA_1P_PASSWORD" --env QA_REMOTE_GRID_USERNAME="$QA_REMOTE_GRID_USERNAME" --env QA_GOOGLE_CDN_SIGNURL_KEY="$QA_GOOGLE_CDN_SIGNURL_KEY" --env QA_TEST_CASE_PROJECT_TOKEN="$QA_TEST_CASE_PROJECT_TOKEN" --env QA_GITHUB_USERNAME="$QA_GITHUB_USERNAME" --env QA_SLOW_CONNECTION_THROUGHPUT_KBPS="$QA_SLOW_CONNECTION_THROUGHPUT_KBPS" --env QA_GITHUB_PASSWORD="$QA_GITHUB_PASSWORD" --env QA_ARTIFACTS_DIR="$QA_ARTIFACTS_DIR" --env QA_GITLAB_TLS_CERTIFICATE="$QA_GITLAB_TLS_CERTIFICATE" --env QA_RUN_TYPE="$QA_RUN_TYPE" --env QA_JIRA_ADMIN_PASSWORD="$QA_JIRA_ADMIN_PASSWORD" --env QA_GITHUB_OAUTH_APP_SECRET="$QA_GITHUB_OAUTH_APP_SECRET" --env QA_PATH="$QA_PATH" --env QA_JIRA_ADMIN_USERNAME="$QA_JIRA_ADMIN_USERNAME" --env QA_ADMIN_ACCESS_TOKEN="$QA_ADMIN_ACCESS_TOKEN" --env QA_COMMAND="$QA_COMMAND" --env QA_EE_LICENSE="$QA_EE_LICENSE" --env QA_1P_EMAIL="$QA_1P_EMAIL" --env QA_EE_ACTIVATION_CODE="$QA_EE_ACTIVATION_CODE" --env QA_GOOGLE_CLIENT_EMAIL="$QA_GOOGLE_CLIENT_EMAIL" --env QA_ALLURE_AWS_SECRET_ACCESS_KEY="$QA_ALLURE_AWS_SECRET_ACCESS_KEY" --env QA_GCS_BUCKET_NAME="$QA_GCS_BUCKET_NAME" --env QA_FACEBOOK_OAUTH_APP_SECRET="$QA_FACEBOOK_OAUTH_APP_SECRET" --env QA_GOOGLE_PROJECT="$QA_GOOGLE_PROJECT" --env QA_KNAPSACK_REPORT_GCS_CREDENTIALS="$QA_KNAPSACK_REPORT_GCS_CREDENTIALS" --env QA_AWS_S3_KEY_ID="$QA_AWS_S3_KEY_ID" --env QA_RUN_ALL_TESTS="$QA_RUN_ALL_TESTS" --env QA_GITLAB_CI_TOKEN="$QA_GITLAB_CI_TOKEN" --env QA_REMOTE_GRID_ACCESS_KEY="$QA_REMOTE_GRID_ACCESS_KEY" --env QA_FRAMEWORK_CHANGES="$QA_FRAMEWORK_CHANGES" --env QA_INTERCEPT_REQUESTS="$QA_INTERCEPT_REQUESTS" --env QA_INFLUXDB_URL="$QA_INFLUXDB_URL" --env QA_THIRD_PARTY_DOCKER_USER="$QA_THIRD_PARTY_DOCKER_USER" --env QA_THIRD_PARTY_DOCKER_PASSWORD="$QA_THIRD_PARTY_DOCKER_PASSWORD" --env QA_1P_SECRET="$QA_1P_SECRET" --env QA_AWS_S3_ACCESS_KEY="$QA_AWS_S3_ACCESS_KEY" --env QA_LOG_LEVEL="$QA_LOG_LEVEL" --env QA_GITHUB_OAUTH_APP_ID="$QA_GITHUB_OAUTH_APP_ID" --env QA_EXPORT_TEST_METRICS="$QA_EXPORT_TEST_METRICS" --env QA_GOOGLE_CDN_LB="$QA_GOOGLE_CDN_LB" --env QA_1P_GITHUB_UUID="$QA_1P_GITHUB_UUID" --env QA_FACEBOOK_PASSWORD="$QA_FACEBOOK_PASSWORD" --env QA_GCS_CDN_BUCKET_NAME="$QA_GCS_CDN_BUCKET_NAME" --env QA_SCENARIO="$QA_SCENARIO" --env QA_GOOGLE_JSON_KEY="$QA_GOOGLE_JSON_KEY" --env QA_SUITES="$QA_SUITES" --env QA_RUN_ALL_E2E_LABEL="$QA_RUN_ALL_E2E_LABEL" --env QA_ALLURE_AWS_ACCESS_KEY_ID="$QA_ALLURE_AWS_ACCESS_KEY_ID" --env QA_AWS_S3_BUCKET_NAME="$QA_AWS_S3_BUCKET_NAME" --env QA_GENERATE_ALLURE_REPORT="$QA_GENERATE_ALLURE_REPORT" --env QA_CAN_TEST_PRAEFECT="$QA_CAN_TEST_PRAEFECT" --env QA_RUN_TESTS_ON_GDK="$QA_RUN_TESTS_ON_GDK" --env QA_INFLUXDB_TOKEN="$QA_INFLUXDB_TOKEN" --env QA_TEST_LICENSE_MODE="$QA_TEST_LICENSE_MODE" --env QA_TEST_SESSION_TOKEN="$QA_TEST_SESSION_TOKEN" --env QA_SLOW_CONNECTION_LATENCY_MS="$QA_SLOW_CONNECTION_LATENCY_MS" --env QA_RESULTS_ISSUE_PROJECT_TOKEN="$QA_RESULTS_ISSUE_PROJECT_TOKEN" --env QA_GITHUB_ACCESS_TOKEN="$QA_GITHUB_ACCESS_TOKEN" --env QA_FACEBOOK_USERNAME="$QA_FACEBOOK_USERNAME" --env QA_OMNIBUS_MR_TESTS="$QA_OMNIBUS_MR_TESTS" --env QA_THIRD_PARTY_DOCKER_REPOSITORY="$QA_THIRD_PARTY_DOCKER_REPOSITORY" --env QA_SELENOID_BROWSER_VERSION="111.0" --env QA_VIDEO_RECORDER_IMAGE="presidenten/selenoid-manual-video-recorder" --env QA_VIDEO_RECORDER_VERSION="latest" --env QA_SELENOID_BROWSER_IMAGE="selenoid/chrome" --env GITLAB_QA_ADMIN_ACCESS_TOKEN="$GITLAB_QA_ADMIN_ACCESS_TOKEN" --env GITLAB_QA_PASSWORD_1="$GITLAB_QA_PASSWORD_1" --env GITLAB_QA_PASSWORD_2="$GITLAB_QA_PASSWORD_2" --env KNAPSACK_GENERATE_REPORT="$KNAPSACK_GENERATE_REPORT" --env RSPEC_FAST_QUARANTINE_PATH="$RSPEC_FAST_QUARANTINE_PATH" --env RSPEC_SKIPPED_TESTS_REPORT_PATH="$RSPEC_SKIPPED_TESTS_REPORT_PATH" --env CI="$CI" --env CI_JOB_NAME="$CI_JOB_NAME" --env CI_JOB_NAME_SLUG="$CI_JOB_NAME_SLUG" --env CI_JOB_URL="$CI_JOB_URL" --env CI_JOB_TOKEN="$CI_JOB_TOKEN" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env CI_PROJECT_NAME="$CI_PROJECT_NAME" --env CI_PROJECT_PATH="$CI_PROJECT_PATH" --env CI_PIPELINE_SOURCE="$CI_PIPELINE_SOURCE" --env CI_PIPELINE_URL="$CI_PIPELINE_URL" --env CI_PIPELINE_CREATED_AT="$CI_PIPELINE_CREATED_AT" --env GITLAB_CI="$GITLAB_CI" --env GITLAB_QA_USER_AGENT="$GITLAB_QA_USER_AGENT" --env GITLAB_INITIAL_ROOT_PASSWORD="5iveL!fe" --env COLORIZED_LOGS="$COLORIZED_LOGS" --env RELEASE="$RELEASE" --env RELEASE_REGISTRY_URL="registry.gitlab.com" --env RELEASE_REGISTRY_USERNAME="gitlab-ci-token" --env RELEASE_REGISTRY_PASSWORD="[MASKED]" --env SELENOID_DIRECTORY="/builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/selenoid" --env SCHEDULE_TYPE="$SCHEDULE_TYPE" --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-11-16-03-31-26-a9b6d50e/gitlab-ee-qa-8549fb76:/home/gitlab/qa/tmp:z --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /builds/gitlab-org/gitlab/qa/rspec:/home/gitlab/qa/rspec:z --name gitlab-ee-qa-8549fb76 registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:7f3e0a47652c47a96f2c5ea8bca5d2ca4c6d8255 Test::Instance::All http://gitlab-gitaly-cluster.test -- --format QA::Support::JsonFormatter --out tmp/rspec-5547762662.json --format RspecJunitFormatter --out tmp/rspec-5547762662.xml --format html --out tmp/rspec-gitaly-cluster.htm --color --format documentation --tag gitaly_cluster`
3322023-11-16 03:38:14 / CONF	:: 
333      ==> Base URL: http://gitlab-gitaly-cluster.test
334      ==> Browser: #<Selenium::WebDriver::Chrome::Driver:0x00007f10b61432f0>
335      ==> Libraries: Chemlab::Vendor
336[Nov 16 2023 03:38:14 UTC (QA Tests)] INFO  -- Caching token for username: root, last six chars of token:sixwPA
337[Nov 16 2023 03:38:14 UTC (QA Tests)] INFO  -- Browser: chrome
338[Nov 16 2023 03:38:14 UTC (QA Tests)] INFO  -- Performing sanity check for environment!
339[Nov 16 2023 03:38:15 UTC (QA Tests)] INFO  -- Performing initial license fabrication!
340[Nov 16 2023 03:38:15 UTC (QA Tests)] INFO  -- Successfully added license key. Details:
341{:test_license_mode=>false, :license_key_length=>2136, :last_five_characters=>"cbiJ9", :plan=>"ultimate", :starts_at=>"2023-02-03", :user_limit=>10000}
342[Nov 16 2023 03:38:15 UTC (QA Tests)] INFO  -- ==> Built a QA::EE::Resource::License with id '1' via api in 0.4 seconds
343[Nov 16 2023 03:38:15 UTC (QA Tests)] INFO  -- Disabling sync with External package metadata database
344[Nov 16 2023 03:38:16 UTC (QA Tests)] INFO  -- Setting application settings: {:package_metadata_purl_types=>[11]}
345Run options:
346  include {:gitaly_cluster=>true}
347  exclude {:geo=>true, :requires_praefect=>true}
348Randomized with seed 40209
349Systems
350  Gitaly Cluster replication queue
351[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly Cluster replication queue allows replication of different repository after interruption
352[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
353[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
354`
355[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
356[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
357[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
358[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
359[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
360[Nov 16 2023 03:39:42 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
361[Nov 16 2023 03:39:43 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
362[Nov 16 2023 03:39:43 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
363[Nov 16 2023 03:39:45 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
364`
365[Nov 16 2023 03:39:45 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
366[Nov 16 2023 03:39:45 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
367[Nov 16 2023 03:39:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
368`
369[Nov 16 2023 03:39:46 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
370[Nov 16 2023 03:39:46 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
371[Nov 16 2023 03:39:47 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
372`
373[Nov 16 2023 03:39:47 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
374[Nov 16 2023 03:39:47 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
375[Nov 16 2023 03:39:47 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
376`
377[Nov 16 2023 03:39:48 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
378`
379[Nov 16 2023 03:39:50 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
380`
381[Nov 16 2023 03:39:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
382`
383[Nov 16 2023 03:39:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
384`
385[Nov 16 2023 03:39:53 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
386`
387[Nov 16 2023 03:39:54 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
388`
389[Nov 16 2023 03:39:55 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
390`
391[Nov 16 2023 03:39:57 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
392`
393[Nov 16 2023 03:39:58 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
394`
395[Nov 16 2023 03:39:58 UTC (QA Tests)] INFO  -- Retrieved cached token for username: root, last six chars of token:sixwPA
396[Nov 16 2023 03:39:58 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::PersonalAccessToken with name 'api-pat-root-y2ukj9dt' via cache in 0.0 seconds
397[Nov 16 2023 03:39:59 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 1.41 seconds
398[Nov 16 2023 03:40:01 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156' via api in 2.89 seconds
399[Nov 16 2023 03:40:04 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156/gitaly_cluster-31c3ef26c1a2a308' via api in 6.2 seconds
400[Nov 16 2023 03:40:30 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
401[Nov 16 2023 03:40:30 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
402[Nov 16 2023 03:40:30 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
403[Nov 16 2023 03:40:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
404`
405[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
406`
407[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
408`
409[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
410`
411[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
412`
413[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
414`
415[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
416`
417[Nov 16 2023 03:40:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
418`
419[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
420`
421[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
422`
423[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
424`
425[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
426`
427[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
428`
429[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
430[Nov 16 2023 03:40:32 UTC (QA Tests)] INFO  -- Executing: `docker pause praefect`
431[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Setting jobs in replication queue to `in_progress` and acquiring locks
432[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"update replication_queue set state = 'in_progress';
433insert into replication_queue_job_lock (job_id, lock_id, triggered_at)
434  select id, rq.lock_id, created_at from replication_queue rq
435    left join replication_queue_job_lock rqjl on rq.id = rqjl.job_id
436    where state = 'in_progress' and rqjl.job_id is null;
437update replication_queue_lock set acquired = 't';
438\""
439`
440[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
441[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Executing: `docker unpause praefect`
442[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} praefect`
443[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
444[Nov 16 2023 03:40:33 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
445[Nov 16 2023 03:40:34 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.06 seconds
446[Nov 16 2023 03:40:34 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156' via api in 0.22 seconds
447[Nov 16 2023 03:40:37 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156/the_awesome_project-262e89d723e02463' via api in 3.1 seconds
448[Nov 16 2023 03:40:37 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[2]"'`
449[Nov 16 2023 03:41:07 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
450[Nov 16 2023 03:41:07 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
451`
452[Nov 16 2023 03:41:07 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
453[Nov 16 2023 03:41:07 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
454[Nov 16 2023 03:41:07 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
455[Nov 16 2023 03:41:07 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
456[Nov 16 2023 03:41:08 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
457[Nov 16 2023 03:41:08 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
458[Nov 16 2023 03:41:09 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
459[Nov 16 2023 03:41:09 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
460[Nov 16 2023 03:41:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
461`
462[Nov 16 2023 03:41:10 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
463[Nov 16 2023 03:41:10 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
464[Nov 16 2023 03:41:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
465`
466[Nov 16 2023 03:41:11 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
467[Nov 16 2023 03:41:11 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
468[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
469`
470[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Clearing the replication queue
471[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from replication_queue_job_lock;
472delete from replication_queue_lock;
473delete from replication_queue;
474\""
475`
476    allows replication of different repository after interruption
477Systems
478  Gitaly backend node recovery
479[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly backend node recovery recovers from dataloss
480[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
481[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
482`
483[Nov 16 2023 03:41:12 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
484[Nov 16 2023 03:41:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
485[Nov 16 2023 03:41:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
486[Nov 16 2023 03:41:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
487[Nov 16 2023 03:41:13 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
488[Nov 16 2023 03:41:13 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
489[Nov 16 2023 03:41:14 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
490[Nov 16 2023 03:41:14 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
491[Nov 16 2023 03:41:15 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
492`
493[Nov 16 2023 03:41:15 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
494[Nov 16 2023 03:41:15 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
495[Nov 16 2023 03:41:16 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
496`
497[Nov 16 2023 03:41:16 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
498[Nov 16 2023 03:41:16 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
499[Nov 16 2023 03:41:17 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
500`
501[Nov 16 2023 03:41:17 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.07 seconds
502[Nov 16 2023 03:41:17 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156' via api in 0.24 seconds
503[Nov 16 2023 03:41:20 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156/gitaly_cluster-5ec7a83c2986ac7c' via api in 2.8 seconds
504[Nov 16 2023 03:41:20 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
505`
506[Nov 16 2023 03:41:20 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`
507[Nov 16 2023 03:41:50 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
508[Nov 16 2023 03:41:50 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly1`
509[Nov 16 2023 03:41:50 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
510`
511[Nov 16 2023 03:41:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
512`
513[Nov 16 2023 03:41:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
514`
515[Nov 16 2023 03:41:53 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
516`
517[Nov 16 2023 03:41:54 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
518`
519[Nov 16 2023 03:41:55 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
520`
521[Nov 16 2023 03:41:57 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
522`
523[Nov 16 2023 03:41:58 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
524`
525[Nov 16 2023 03:41:59 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
526`
527[Nov 16 2023 03:42:00 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
528`
529[Nov 16 2023 03:42:00 UTC (QA Tests)] INFO  -- Waiting for health check failure on gitaly1
530[Nov 16 2023 03:42:00 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
531`
532[Nov 16 2023 03:42:02 UTC (QA Tests)] INFO  -- QA::Resource::Project - wait_for_push with commit message "pushed after failover"
533[Nov 16 2023 03:42:02 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.17 seconds
534[Nov 16 2023 03:42:02 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select job from replication_queue
535where state = 'ready'
536  and job ->> 'change' = 'update'
537  and job ->> 'target_node_storage' = 'gitaly1';
538\""
539`
540[Nov 16 2023 03:42:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
541[Nov 16 2023 03:42:02 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly1`
542[Nov 16 2023 03:42:03 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
543[Nov 16 2023 03:42:03 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
544[Nov 16 2023 03:42:03 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
545[Nov 16 2023 03:42:04 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
546`
547[Nov 16 2023 03:42:04 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
548[Nov 16 2023 03:42:04 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
549[Nov 16 2023 03:42:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
550`
551[Nov 16 2023 03:42:05 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
552[Nov 16 2023 03:42:05 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
553[Nov 16 2023 03:42:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
554`
555[Nov 16 2023 03:42:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
556`
557[Nov 16 2023 03:42:06 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`
558[Nov 16 2023 03:42:36 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
559[Nov 16 2023 03:42:36 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
560[Nov 16 2023 03:42:36 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
561`
562[Nov 16 2023 03:42:37 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
563`
564[Nov 16 2023 03:42:38 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
565`
566[Nov 16 2023 03:42:40 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
567`
568[Nov 16 2023 03:42:41 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
569`
570[Nov 16 2023 03:42:42 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
571`
572[Nov 16 2023 03:42:43 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
573`
574[Nov 16 2023 03:42:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
575`
576[Nov 16 2023 03:42:45 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
577`
578[Nov 16 2023 03:42:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
579`
580[Nov 16 2023 03:42:46 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
581[Nov 16 2023 03:42:46 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly3`
582[Nov 16 2023 03:42:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
583`
584[Nov 16 2023 03:42:48 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
585`
586[Nov 16 2023 03:42:49 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
587`
588[Nov 16 2023 03:42:50 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
589`
590[Nov 16 2023 03:42:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
591`
592[Nov 16 2023 03:42:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
593`
594[Nov 16 2023 03:42:53 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
595`
596[Nov 16 2023 03:42:54 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
597`
598[Nov 16 2023 03:42:55 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
599`
600[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
601`
602[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
603[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
604`
605[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
606[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
607[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
608[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
609[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
610[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly3`
611[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
612[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
613[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
614[Nov 16 2023 03:42:57 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
615[Nov 16 2023 03:42:58 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
616[Nov 16 2023 03:42:58 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
617[Nov 16 2023 03:42:59 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
618`
619[Nov 16 2023 03:42:59 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
620[Nov 16 2023 03:42:59 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
621[Nov 16 2023 03:43:00 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
622`
623[Nov 16 2023 03:43:00 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
624[Nov 16 2023 03:43:00 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
625[Nov 16 2023 03:43:01 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
626`
627    recovers from dataloss
628Systems
629  with Gitaly automatic failover and recovery
630[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
631[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
632`
633[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
634[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
635[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
636[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
637[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
638[Nov 16 2023 03:43:02 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
639[Nov 16 2023 03:43:03 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
640[Nov 16 2023 03:43:03 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
641[Nov 16 2023 03:43:04 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
642`
643[Nov 16 2023 03:43:04 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
644[Nov 16 2023 03:43:04 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
645[Nov 16 2023 03:43:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
646`
647[Nov 16 2023 03:43:05 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
648[Nov 16 2023 03:43:05 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
649[Nov 16 2023 03:43:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
650`
651[Nov 16 2023 03:43:07 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.07 seconds
652[Nov 16 2023 03:43:07 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156' via api in 0.22 seconds
653[Nov 16 2023 03:43:08 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156/gitaly_cluster-dba3d78485188b61' via api in 1.69 seconds
654[Nov 16 2023 03:43:08 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
655`
656[Nov 16 2023 03:43:08 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
657[Nov 16 2023 03:43:38 UTC (QA Tests)] INFO  -- Starting test: Systems with Gitaly automatic failover and recovery automatically fails over
658[Nov 16 2023 03:43:38 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
659[Nov 16 2023 03:43:38 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
660[Nov 16 2023 03:43:38 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
661`
662[Nov 16 2023 03:43:39 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
663`
664[Nov 16 2023 03:43:40 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
665`
666[Nov 16 2023 03:43:41 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
667`
668[Nov 16 2023 03:43:42 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
669`
670[Nov 16 2023 03:43:43 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
671`
672[Nov 16 2023 03:43:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
673`
674[Nov 16 2023 03:43:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
675`
676[Nov 16 2023 03:43:47 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
677`
678[Nov 16 2023 03:43:48 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
679`
680[Nov 16 2023 03:43:48 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
681[Nov 16 2023 03:43:48 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly3`
682[Nov 16 2023 03:43:48 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
683`
684[Nov 16 2023 03:43:49 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
685`
686[Nov 16 2023 03:43:50 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
687`
688[Nov 16 2023 03:43:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
689`
690[Nov 16 2023 03:43:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
691`
692[Nov 16 2023 03:43:54 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
693`
694[Nov 16 2023 03:43:55 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
695`
696[Nov 16 2023 03:43:56 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
697`
698[Nov 16 2023 03:43:57 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
699`
700[Nov 16 2023 03:43:58 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
701`
702[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- QA::Resource::Project - wait_for_push with commit message "first_added_commit_message to primary gitaly node"
703[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 1.84 seconds
704[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
705[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
706`
707[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
708[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
709[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
710[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
711[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
712[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly3`
713[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
714[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
715[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
716[Nov 16 2023 03:44:00 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
717[Nov 16 2023 03:44:01 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
718[Nov 16 2023 03:44:01 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
719[Nov 16 2023 03:44:02 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
720`
721[Nov 16 2023 03:44:03 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
722[Nov 16 2023 03:44:03 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
723[Nov 16 2023 03:44:04 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
724`
725[Nov 16 2023 03:44:04 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
726[Nov 16 2023 03:44:04 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
727[Nov 16 2023 03:44:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
728`
729[Nov 16 2023 03:44:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
730`
731[Nov 16 2023 03:44:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
732`
733[Nov 16 2023 03:44:06 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
734[Nov 16 2023 03:44:36 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
735[Nov 16 2023 03:44:36 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly1`
736[Nov 16 2023 03:44:36 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
737`
738[Nov 16 2023 03:44:37 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
739`
740[Nov 16 2023 03:44:38 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
741`
742[Nov 16 2023 03:44:40 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
743`
744[Nov 16 2023 03:44:41 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
745`
746[Nov 16 2023 03:44:42 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
747`
748[Nov 16 2023 03:44:43 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
749`
750[Nov 16 2023 03:44:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
751`
752[Nov 16 2023 03:44:45 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
753`
754[Nov 16 2023 03:44:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
755`
756[Nov 16 2023 03:44:46 UTC (QA Tests)] INFO  -- Waiting for health check failure on gitaly1
757[Nov 16 2023 03:44:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
758`
759    automatically fails over
760    when recovering from dataloss after failover
761[Nov 16 2023 03:44:48 UTC (QA Tests)] INFO  -- Starting test: Systems with Gitaly automatic failover and recovery when recovering from dataloss after failover automatically reconciles
762[Nov 16 2023 03:44:48 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
763[Nov 16 2023 03:44:48 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly1`
764[Nov 16 2023 03:44:48 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
765[Nov 16 2023 03:44:48 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
766[Nov 16 2023 03:44:48 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
767[Nov 16 2023 03:44:49 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
768`
769[Nov 16 2023 03:44:49 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
770[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
771[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
772`
773[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
774[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
775[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
776[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
777[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
778[Nov 16 2023 03:45:19 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
779[Nov 16 2023 03:45:20 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
780[Nov 16 2023 03:45:20 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
781[Nov 16 2023 03:45:21 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
782`
783[Nov 16 2023 03:45:21 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
784[Nov 16 2023 03:45:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
785[Nov 16 2023 03:45:22 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
786`
787[Nov 16 2023 03:45:22 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
788[Nov 16 2023 03:45:22 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
789[Nov 16 2023 03:45:23 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
790`
791      automatically reconciles
792Systems
793  Gitaly distributed reads
794[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly distributed reads reads from each node
795[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
796[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
797`
798[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
799[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
800[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
801[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
802[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
803[Nov 16 2023 03:45:24 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
804[Nov 16 2023 03:45:25 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
805[Nov 16 2023 03:45:25 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
806[Nov 16 2023 03:45:26 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
807`
808[Nov 16 2023 03:45:26 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
809[Nov 16 2023 03:45:26 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
810[Nov 16 2023 03:45:27 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
811`
812[Nov 16 2023 03:45:27 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
813[Nov 16 2023 03:45:27 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
814[Nov 16 2023 03:45:28 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
815`
816[Nov 16 2023 03:45:28 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.06 seconds
817[Nov 16 2023 03:45:28 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156' via api in 0.17 seconds
818[Nov 16 2023 03:45:30 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156/gitaly_cluster-33fab77ddcb69448' via api in 2.33 seconds
819[Nov 16 2023 03:45:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
820`
821[Nov 16 2023 03:45:31 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[5]"'`
822[Nov 16 2023 03:46:00 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
823[Nov 16 2023 03:46:00 UTC (QA Tests)] INFO  -- Reading from the repository
824[Nov 16 2023 03:46:01 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
825[Nov 16 2023 03:46:06 UTC (QA Tests)] INFO  -- Reading from the repository
826[Nov 16 2023 03:46:07 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
827[Nov 16 2023 03:46:12 UTC (QA Tests)] INFO  -- Reading from the repository
828[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
829[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
830    reads from each node
831    when a node is unhealthy
832[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly distributed reads when a node is unhealthy does not read from the unhealthy node
833[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
834[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
835`
836[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
837[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
838[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
839[Nov 16 2023 03:46:13 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
840[Nov 16 2023 03:46:14 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
841[Nov 16 2023 03:46:14 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
842[Nov 16 2023 03:46:15 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
843[Nov 16 2023 03:46:15 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
844[Nov 16 2023 03:46:16 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
845`
846[Nov 16 2023 03:46:16 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
847[Nov 16 2023 03:46:16 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
848[Nov 16 2023 03:46:17 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
849`
850[Nov 16 2023 03:46:17 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
851[Nov 16 2023 03:46:17 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
852[Nov 16 2023 03:46:18 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
853`
854[Nov 16 2023 03:46:18 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.06 seconds
855[Nov 16 2023 03:46:18 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156' via api in 0.17 seconds
856[Nov 16 2023 03:46:21 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-11-16-03-39-58-e5aa0b0fff0e6156/gitaly_cluster-afc57b37a0d5ecb5' via api in 2.98 seconds
857[Nov 16 2023 03:46:21 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
858`
859[Nov 16 2023 03:46:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[6]"'`
860[Nov 16 2023 03:46:50 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
861[Nov 16 2023 03:46:50 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
862[Nov 16 2023 03:46:50 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
863`
864[Nov 16 2023 03:46:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
865`
866[Nov 16 2023 03:46:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
867`
868[Nov 16 2023 03:46:54 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
869`
870[Nov 16 2023 03:46:55 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
871`
872[Nov 16 2023 03:46:56 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
873`
874[Nov 16 2023 03:46:57 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
875`
876[Nov 16 2023 03:46:58 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
877`
878[Nov 16 2023 03:46:59 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
879`
880[Nov 16 2023 03:47:00 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
881`
882[Nov 16 2023 03:47:00 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
883[Nov 16 2023 03:47:00 UTC (QA Tests)] INFO  -- Reading from the repository
884[Nov 16 2023 03:47:06 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
885[Nov 16 2023 03:47:07 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
886[Nov 16 2023 03:47:08 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
887[Nov 16 2023 03:47:09 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
888[Nov 16 2023 03:47:09 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
889[Nov 16 2023 03:47:09 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
890[Nov 16 2023 03:47:09 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
891[Nov 16 2023 03:47:09 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
892      does not read from the unhealthy node
893Finished in 8 minutes 52 seconds (files took 4.94 seconds to load)
8946 examples, 0 failures
895Randomized with seed 40209
896[Nov 16 2023 03:47:10 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select n_live_tup, n_dead_tup, relname from pg_stat_all_tables order by n_live_tup DESC, n_dead_tup DESC;'"`
897[Nov 16 2023 03:47:10 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select * from pg_stat_user_tables;'"`
898[Nov 16 2023 03:47:10 UTC (Gitlab QA)] INFO  -- Shell command: `docker ps -f name=gitlab-gitaly-cluster`
899[Nov 16 2023 03:47:10 UTC (Gitlab QA)] INFO  -- Shell command: `docker rm -f gitlab-gitaly-cluster`
 900  Running after_script  
  00:23
901Running after script...
902$ if [ "$CI_JOB_STATUS" == "failed" ]; then # collapsed multi-line command
903$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
904$ echo -e "\e[0Ksection_start:`date +%s`:report_results_section[collapsed=true]\r\e[0KReport results"
 905  Report results  
  00:23
 927  Saving cache for successful job  
  00:01
928Not uploading cache qa-e2e-ruby-3.0-5fe14f76d4eb79aabf540b0a58e6de08-16 due to policy
 929  Uploading artifacts for successful job  
  00:03
930Uploading artifacts...
931gitlab-qa-run-*: found 219 matching artifact files and directories 
932Uploading artifacts as "archive" to coordinator... 201 Created  id=5547762662 responseStatus=201 Created token=64_AaMvy
933Uploading artifacts...
934gitlab-qa-run-*/**/rspec-*.xml: found 1 matching artifact files and directories 
935Uploading artifacts as "junit" to coordinator... 201 Created  id=5547762662 responseStatus=201 Created token=64_AaMvy
936Uploading artifacts...
937/builds/gitlab-org/gitlab/suite_status.env: found 1 matching artifact files and directories 
938Uploading artifacts as "dotenv" to coordinator... 201 Created  id=5547762662 responseStatus=201 Created token=64_AaMvy
 939  Cleaning up project directory and file based variables  
  00:01
940Job succeeded
Tags: e2e
Job artifacts
The artifacts were removed 1 year ago
Commit 7f3e0a47
Merge branch 'update-usage' into 'master'
Related jobs

instance 1/5
instance 3/5
_quarantine
cloud-activation
decomposition-multiple-db 1/5
decomposition-multiple-db 2/5
decomposition-multiple-db 3/5
decomposition-multiple-db 4/5
decomposition-multiple-db 5/5
decomposition-single-db 1/5
decomposition-single-db 2/5
decomposition-single-db 3/5
decomposition-single-db 4/5
decomposition-single-db 5/5
elasticsearch
gitaly-cluster
gitlab-pages
group-saml
importers
instance 2/5
instance 4/5
instance 5/5
instance-saml
integrations
jira
large-setup
ldap-no-server
ldap-no-tls
ldap-tls
mattermost
metrics
mtls
oauth
object-storage 1/2
object-storage 2/2
object-storage-aws 1/2
object-storage-aws 2/2
object-storage-gcs 1/2
object-storage-gcs 2/2
praefect 1/2
praefect 2/2
registry
registry-with-cdn
repository-storage
service-ping-disabled
smtp
update-major
update-minor