gitaly-cluster

Passed Started 1 year ago by
1Running with gitlab-runner 15.11.0 (436955cb)
2  on gitlab-org-qa-runner ss7o3c7p, system ID: s_6a7379124369
 3  Resolving secrets  
  00:00
 4  Preparing the "docker+machine" executor  
  01:11
5Using Docker executor with image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 ...
6Starting service docker:23.0.1-dind ...
7Pulling docker image docker:23.0.1-dind ...
8Using docker image sha256:0965b14b96d892eee2a963801769af5328817a64ed1af67ec4d76364ddd62706 for docker:23.0.1-dind with digest docker@sha256:d9a0fd8bdd156aae6e9be5db0264e0bc99e000e89e92055d4567e27ff13daff0 ...
9Waiting for services to be up and running (timeout 30 seconds)...
10Authenticating with credentials from job payload (GitLab Registry)
11Pulling docker image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 ...
12Using docker image sha256:f5a02bf774aef64cc33176a18c75da35bad4d7ec9e4ba0e202ce873eb1cf0e76 for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:cfd42d69a6aa3fb6ec4a7e9abb60d75a4a8374b6e6e5e331022117152aef1e99 ...
 13  Preparing environment  
  00:01
14Running on runner-ss7o3c7p-project-278964-concurrent-0 via runner-ss7o3c7p-gitlab-org-qa-1690529966-583903b2-gcp...
 15  Getting source from Git repository  
  00:21
16Fetching changes with git depth set to 20...
17Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/
18Created fresh repository.
19remote: Enumerating objects: 194346, done.        
20remote: Counting objects: 100% (194346/194346), done.        
21remote: Compressing objects: 100% (115006/115006), done.        
22remote: Total 194346 (delta 103604), reused 137133 (delta 72406), pack-reused 0        
23Receiving objects: 100% (194346/194346), 142.39 MiB | 45.24 MiB/s, done.
24Resolving deltas: 100% (103604/103604), done.
25From https://gitlab.com/gitlab-org/gitlab
26 * [new ref]             refs/pipelines/948364738 -> refs/pipelines/948364738
27 * [new branch]          master                   -> origin/master
28Checking out 09612fea as detached HEAD (ref is master)...
29Skipping Git submodules setup
 30  Restoring cache  
  00:04
31Checking cache for qa-e2e-ruby-3.0-c62e56e5e86789fd869c3bcac450f355-16...
32Downloading cache.zip from https://storage.googleapis.com/gitlab-qa-runner-cache-1/project/278964/qa-e2e-ruby-3.0-c62e56e5e86789fd869c3bcac450f355-16 
33WARNING: .gems/gems/deprecation_toolkit-2.0.3/gemfiles/test/deprecations: chmod .gems/gems/deprecation_toolkit-2.0.3/gemfiles/test/deprecations: no such file or directory (suppressing repeats) 
34Successfully extracted cache
 35  Downloading artifacts  
  00:01
36Downloading artifacts for download-knapsack-report (4758981991)...
37Downloading artifacts from coordinator... ok        host=storage.googleapis.com id=4758981991 responseStatus=200 OK token=64_QuM-m
 38  Executing "step_script" stage of the job script  
  13:35
39Using docker image sha256:f5a02bf774aef64cc33176a18c75da35bad4d7ec9e4ba0e202ce873eb1cf0e76 for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:cfd42d69a6aa3fb6ec4a7e9abb60d75a4a8374b6e6e5e331022117152aef1e99 ...
40$ echo "SUITE_RAN=true" > $QA_SUITE_STATUS_ENV_FILE
41$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
42$ if [ "$RUN_WITH_BUNDLE" == "true" ]; then # collapsed multi-line command
43Bundle complete! 37 Gemfile dependencies, 135 gems now installed.
44Use `bundle info [gemname]` to see where a bundled gem is installed.
45$ export QA_COMMAND="$BUNDLE_PREFIX gitlab-qa ${QA_SCENARIO:=Test::Instance::Image} $RELEASE $GITLAB_QA_OPTS -- $QA_TESTS $QA_RSPEC_TAGS $RSPEC_REPORT_OPTS"
46$ echo "Running - '$QA_COMMAND'"
47Running - 'bundle exec gitlab-qa Test::Integration::GitalyCluster registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898  --   --format QA::Support::JsonFormatter --out tmp/rspec-4758982049.json --format RspecJunitFormatter --out tmp/rspec-4758982049.xml --format html --out tmp/rspec-$(echo gitaly-cluster | sed 's/[ /]/_/g').htm --color --format documentation'
48$ eval "$QA_COMMAND"
49[Jul 28 2023 08:06:20 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q postgres:11`
50[Jul 28 2023 08:06:24 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^postgres$`
51[Jul 28 2023 08:06:24 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
52[Jul 28 2023 08:06:24 UTC (Gitlab QA)] INFO  -- Shell command: `docker network create test`
53[Jul 28 2023 08:06:24 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name postgres --net test --env POSTGRES_PASSWORD="SQL_PASSWORD" postgres:11`
54[Jul 28 2023 08:06:26 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`
55[Jul 28 2023 08:06:26 UTC (Gitlab QA)] ERROR -- Shell command output:
56psql: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
57	Is the server running locally and accepting connections on that socket?
58[Jul 28 2023 08:06:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`
59[Jul 28 2023 08:06:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec postgres bash -c "psql -U postgres -d template1 -c \"CREATE DATABASE praefect_production OWNER postgres\""`
60[Jul 28 2023 08:06:32 UTC (Gitlab QA)] INFO  -- Starting Gitaly Cluster
61[Jul 28 2023 08:06:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^alpine-755aa0e6$`
62[Jul 28 2023 08:06:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
63[Jul 28 2023 08:06:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name alpine-755aa0e6 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z alpine:latest tail -f /dev/null`
64[Jul 28 2023 08:06:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
65[Jul 28 2023 08:06:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.2.1/tls_certificates/authority/. alpine-755aa0e6:/etc/gitlab/trusted-certs`
66[Jul 28 2023 08:06:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.2.1/tls_certificates/gitlab/. alpine-755aa0e6:/etc/gitlab/ssl`
67[Jul 28 2023 08:06:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker ps -f name=alpine-755aa0e6`
68[Jul 28 2023 08:06:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker rm -f alpine-755aa0e6`
69[Jul 28 2023 08:06:45 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
70[Jul 28 2023 08:06:46 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
71[Jul 28 2023 08:07:05 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitlab-gitaly-cluster$`
72[Jul 28 2023 08:07:05 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly1$`
73[Jul 28 2023 08:07:05 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
74[Jul 28 2023 08:07:05 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
75[Jul 28 2023 08:07:05 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitaly1 --net test --hostname gitaly1.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/gitaly1/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
76[Jul 28 2023 08:07:05 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitlab-gitaly-cluster --net test --hostname gitlab-gitaly-cluster.test -p 80 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/gitlab-gitaly-cluster/logs:/var/log/gitlab:Z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
77[Jul 28 2023 08:07:25 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo \"# Generated by GitLab QA Omnibus Configurator at 2023-07-28 08:06:20
78# Gitlab::QA::Runtime::OmnibusConfigurations::Default
79gitlab_rails['gitlab_default_theme'] = 10 # Light Red Theme
80gitlab_rails['gitlab_disable_animations'] = true # Disable animations
81gitlab_rails['application_settings_cache_seconds'] = 0 # Settings cache expiry
82gitlab_rails['initial_root_password'] = '5iveL!fe' # Initial root password
83external_url 'http://gitlab-gitaly-cluster.test';
84git_data_dirs({
85  'default' => {
86    'gitaly_address' => 'tcp://praefect.test:2305',
87    'gitaly_token' => 'PRAEFECT_EXTERNAL_TOKEN'
88  }
89});
90gitaly['enable'] = false;
91gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
92prometheus['scrape_configs'] = [
93  {
94    'job_name' => 'praefect',
95    'static_configs' => [
96      'targets' => [
97        'praefect.test:9652'
98      ]
99    ]
100  },
101  {
102    'job_name' => 'praefect-gitaly',
103    'static_configs' => [
104      'targets' => [
105        'gitaly1.test:9236',
106        'gitaly2.test:9236',
107        'gitaly3.test:9236'
108      ]
109    ]
110  }
111];
112gitlab_rails['env'] = {'GITLAB_LICENSE_MODE'=>'test', 'CUSTOMER_PORTAL_URL'=>'https://customers.staging.gitlab.com', 'GITLAB_QA_USER_AGENT'=>'*****', 'GITLAB_ALLOW_SEPARATE_CI_DATABASE'=>'true'}\" > /etc/gitlab/gitlab.rb;"`
113[Jul 28 2023 08:07:25 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "echo \"postgresql['enable'] = false;
114redis['enable'] = false;
115nginx['enable'] = false;
116puma['enable'] = false;
117sidekiq['enable'] = false;
118gitlab_workhorse['enable'] = false;
119gitlab_rails['rake_cache_clear'] = false;
120gitlab_rails['auto_migrate'] = false;
121gitlab_exporter['enable'] = false;
122gitlab_kas['enable'] = false;
123praefect['enable'] = false;
124prometheus['enable'] = true;
125gitaly['enable'] = true;
126gitaly['configuration'] = {
127  'listen_addr': '0.0.0.0:8075',
128  'prometheus_listen_addr': '0.0.0.0:9236',
129  'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
130}
131gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
132gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
133git_data_dirs({
134  'gitaly1' => {
135    'path' => '/var/opt/gitlab/git-data'
136  },
137  'gitaly2' => {
138    'path' => '/var/opt/gitlab/git-data'
139  },
140  'gitaly3' => {
141    'path' => '/var/opt/gitlab/git-data'
142  }
143});
144\" > /etc/gitlab/gitlab.rb;"`
145[Jul 28 2023 08:07:25 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitlab-gitaly-cluster`
146[Jul 28 2023 08:07:25 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitaly1`
147[Jul 28 2023 08:08:07 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
148[Jul 28 2023 08:08:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "echo '#!/usr/bin/env bash
149if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
150  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
151  exit 1
152fi
153' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
154[Jul 28 2023 08:08:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly1 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
155[Jul 28 2023 08:08:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
156[Jul 28 2023 08:08:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
157[Jul 28 2023 08:08:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly2$`
158[Jul 28 2023 08:08:08 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
159[Jul 28 2023 08:08:09 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitaly2 --net test --hostname gitaly2.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/gitaly2/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
160[Jul 28 2023 08:08:10 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "echo \"postgresql['enable'] = false;
161redis['enable'] = false;
162nginx['enable'] = false;
163puma['enable'] = false;
164sidekiq['enable'] = false;
165gitlab_workhorse['enable'] = false;
166gitlab_rails['rake_cache_clear'] = false;
167gitlab_rails['auto_migrate'] = false;
168gitlab_exporter['enable'] = false;
169gitlab_kas['enable'] = false;
170praefect['enable'] = false;
171prometheus['enable'] = true;
172gitaly['enable'] = true;
173gitaly['configuration'] = {
174  'listen_addr': '0.0.0.0:8075',
175  'prometheus_listen_addr': '0.0.0.0:9236',
176  'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
177}
178gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
179gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
180git_data_dirs({
181  'gitaly1' => {
182    'path' => '/var/opt/gitlab/git-data'
183  },
184  'gitaly2' => {
185    'path' => '/var/opt/gitlab/git-data'
186  },
187  'gitaly3' => {
188    'path' => '/var/opt/gitlab/git-data'
189  }
190});
191\" > /etc/gitlab/gitlab.rb;"`
192[Jul 28 2023 08:08:11 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitaly2`
193[Jul 28 2023 08:08:53 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
194[Jul 28 2023 08:08:54 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "echo '#!/usr/bin/env bash
195if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
196  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
197  exit 1
198fi
199' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
200[Jul 28 2023 08:08:54 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly2 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
201[Jul 28 2023 08:08:54 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
202[Jul 28 2023 08:08:54 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
203[Jul 28 2023 08:08:54 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly3$`
204[Jul 28 2023 08:08:54 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
205[Jul 28 2023 08:08:55 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name gitaly3 --net test --hostname gitaly3.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/gitaly3/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
206[Jul 28 2023 08:08:57 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "echo \"postgresql['enable'] = false;
207redis['enable'] = false;
208nginx['enable'] = false;
209puma['enable'] = false;
210sidekiq['enable'] = false;
211gitlab_workhorse['enable'] = false;
212gitlab_rails['rake_cache_clear'] = false;
213gitlab_rails['auto_migrate'] = false;
214gitlab_exporter['enable'] = false;
215gitlab_kas['enable'] = false;
216praefect['enable'] = false;
217prometheus['enable'] = true;
218gitaly['enable'] = true;
219gitaly['configuration'] = {
220  'listen_addr': '0.0.0.0:8075',
221  'prometheus_listen_addr': '0.0.0.0:9236',
222  'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
223}
224gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
225gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
226git_data_dirs({
227  'gitaly1' => {
228    'path' => '/var/opt/gitlab/git-data'
229  },
230  'gitaly2' => {
231    'path' => '/var/opt/gitlab/git-data'
232  },
233  'gitaly3' => {
234    'path' => '/var/opt/gitlab/git-data'
235  }
236});
237\" > /etc/gitlab/gitlab.rb;"`
238[Jul 28 2023 08:08:57 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitaly3`
239[Jul 28 2023 08:09:40 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
240[Jul 28 2023 08:09:40 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "echo '#!/usr/bin/env bash
241if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
242  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
243  exit 1
244fi
245' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
246[Jul 28 2023 08:09:40 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitaly3 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
247[Jul 28 2023 08:09:40 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
248[Jul 28 2023 08:09:41 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
249[Jul 28 2023 08:09:41 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^praefect$`
250[Jul 28 2023 08:09:41 UTC (Gitlab QA)] INFO  -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
251[Jul 28 2023 08:09:41 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name praefect --net test --hostname praefect.test -p 2305 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/praefect/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:09612fea8bceb47ca2935844f4a8e1855a3f5898`
252[Jul 28 2023 08:09:43 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec praefect bash -c "echo \"postgresql['enable'] = false;
253redis['enable'] = false;
254nginx['enable'] = false;
255puma['enable'] = false;
256sidekiq['enable'] = false;
257gitlab_workhorse['enable'] = false;
258gitlab_rails['rake_cache_clear'] = false;
259gitlab_rails['auto_migrate'] = false;
260gitlab_exporter['enable'] = false;
261gitlab_kas['enable'] = false;
262gitaly['enable'] = false;
263prometheus['enable'] = true;
264praefect['enable'] = true;
265praefect['configuration'] = {
266  listen_addr: '0.0.0.0:2305',
267  prometheus_listen_addr: '0.0.0.0:9652',
268  auth: {
269    token: 'PRAEFECT_EXTERNAL_TOKEN'
270  },
271  reconciliation: {
272    scheduling_interval: '10s'
273  },
274  database: {
275    host: 'postgres.test',
276    port: 5432,
277    user: 'postgres',
278    password: 'SQL_PASSWORD',
279    dbname: 'praefect_production',
280    sslmode: 'disable'
281  },
282  virtual_storage: [
283    {
284      name: 'default',
285      node: [
286        {
287          'storage': 'gitaly1',
288          'address': 'tcp://gitaly1.test:8075',
289          'token': 'PRAEFECT_INTERNAL_TOKEN'
290        },
291        {
292          'storage': 'gitaly2',
293          'address': 'tcp://gitaly2.test:8075',
294          'token': 'PRAEFECT_INTERNAL_TOKEN'
295        },
296        {
297          'storage': 'gitaly3',
298          'address': 'tcp://gitaly3.test:8075',
299          'token': 'PRAEFECT_INTERNAL_TOKEN'
300        }
301      ],
302    }
303  ]
304}
305\" > /etc/gitlab/gitlab.rb;"`
306[Jul 28 2023 08:09:43 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false praefect`
307[Jul 28 2023 08:10:26 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec praefect bash -c "praefect -config /var/opt/gitlab/praefect/cluster_config.toml check || true"`
308[Jul 28 2023 08:10:26 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' praefect`
309[Jul 28 2023 08:10:26 UTC (Gitlab QA)] INFO  -- Gitaly Cluster Ready
310[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- Waiting for GitLab to become healthy ...
311[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-gitaly-cluster`
312[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- Shell command: `docker port gitlab-gitaly-cluster 80/tcp`
313[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- -> GitLab is available at `http://docker:32768`!
314[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
315[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo '#!/usr/bin/env bash
316if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
317  echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
318  exit 1
319fi
320' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
321[Jul 28 2023 08:10:49 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
322[Jul 28 2023 08:10:50 UTC (Gitlab QA)] INFO  -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.2.1/support/data gitlab-gitaly-cluster:/tmp/data-seeds`
323[Jul 28 2023 08:10:50 UTC (Gitlab QA)] INFO  -- Running exec_commands...
324[Jul 28 2023 08:10:50 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-rails runner /tmp/data-seeds/admin_access_token_seed.rb"`
325[Jul 28 2023 08:11:38 UTC (Gitlab QA)] INFO  -- Running Gitaly Cluster specs!
326[Jul 28 2023 08:11:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
327[Jul 28 2023 08:11:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:09612fea8bceb47ca2935844f4a8e1855a3f5898`
328[Jul 28 2023 08:12:28 UTC (Gitlab QA)] INFO  -- Running test suite `Test::Instance::All` for gitlab-ee
329[Jul 28 2023 08:12:28 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -t --rm --net=test --env QA_AWS_S3_REGION="$QA_AWS_S3_REGION" --env QA_FACEBOOK_OAUTH_APP_ID="$QA_FACEBOOK_OAUTH_APP_ID" --env QA_GOOGLE_CDN_JSON_KEY="$QA_GOOGLE_CDN_JSON_KEY" --env QA_SUITE_STATUS_ENV_FILE="$QA_SUITE_STATUS_ENV_FILE" --env QA_THIRD_PARTY_DOCKER_REGISTRY="$QA_THIRD_PARTY_DOCKER_REGISTRY" --env QA_SAVE_TEST_METRICS="$QA_SAVE_TEST_METRICS" --env QA_GOOGLE_CDN_SIGNURL_KEY_NAME="$QA_GOOGLE_CDN_SIGNURL_KEY_NAME" --env QA_1P_PASSWORD="$QA_1P_PASSWORD" --env QA_REMOTE_GRID_USERNAME="$QA_REMOTE_GRID_USERNAME" --env QA_GOOGLE_CDN_SIGNURL_KEY="$QA_GOOGLE_CDN_SIGNURL_KEY" --env QA_TEST_CASE_PROJECT_TOKEN="$QA_TEST_CASE_PROJECT_TOKEN" --env QA_GITHUB_USERNAME="$QA_GITHUB_USERNAME" --env QA_SLOW_CONNECTION_THROUGHPUT_KBPS="$QA_SLOW_CONNECTION_THROUGHPUT_KBPS" --env QA_GITHUB_PASSWORD="$QA_GITHUB_PASSWORD" --env QA_ARTIFACTS_DIR="$QA_ARTIFACTS_DIR" --env QA_GITLAB_TLS_CERTIFICATE="$QA_GITLAB_TLS_CERTIFICATE" --env QA_RUN_TYPE="$QA_RUN_TYPE" --env QA_JIRA_ADMIN_PASSWORD="$QA_JIRA_ADMIN_PASSWORD" --env QA_GITHUB_OAUTH_APP_SECRET="$QA_GITHUB_OAUTH_APP_SECRET" --env QA_PATH="$QA_PATH" --env QA_JIRA_ADMIN_USERNAME="$QA_JIRA_ADMIN_USERNAME" --env QA_ADMIN_ACCESS_TOKEN="$QA_ADMIN_ACCESS_TOKEN" --env QA_COMMAND="$QA_COMMAND" --env QA_EE_LICENSE="$QA_EE_LICENSE" --env QA_1P_EMAIL="$QA_1P_EMAIL" --env QA_EE_ACTIVATION_CODE="$QA_EE_ACTIVATION_CODE" --env QA_GOOGLE_CLIENT_EMAIL="$QA_GOOGLE_CLIENT_EMAIL" --env QA_ALLURE_AWS_SECRET_ACCESS_KEY="$QA_ALLURE_AWS_SECRET_ACCESS_KEY" --env QA_GCS_BUCKET_NAME="$QA_GCS_BUCKET_NAME" --env QA_FACEBOOK_OAUTH_APP_SECRET="$QA_FACEBOOK_OAUTH_APP_SECRET" --env QA_GOOGLE_PROJECT="$QA_GOOGLE_PROJECT" --env QA_KNAPSACK_REPORT_GCS_CREDENTIALS="$QA_KNAPSACK_REPORT_GCS_CREDENTIALS" --env QA_AWS_S3_KEY_ID="$QA_AWS_S3_KEY_ID" --env QA_RUN_ALL_TESTS="$QA_RUN_ALL_TESTS" --env QA_GITLAB_CI_TOKEN="$QA_GITLAB_CI_TOKEN" --env QA_REMOTE_GRID_ACCESS_KEY="$QA_REMOTE_GRID_ACCESS_KEY" --env QA_FRAMEWORK_CHANGES="$QA_FRAMEWORK_CHANGES" --env QA_INTERCEPT_REQUESTS="$QA_INTERCEPT_REQUESTS" --env QA_INFLUXDB_URL="$QA_INFLUXDB_URL" --env QA_THIRD_PARTY_DOCKER_USER="$QA_THIRD_PARTY_DOCKER_USER" --env QA_THIRD_PARTY_DOCKER_PASSWORD="$QA_THIRD_PARTY_DOCKER_PASSWORD" --env QA_1P_SECRET="$QA_1P_SECRET" --env QA_AWS_S3_ACCESS_KEY="$QA_AWS_S3_ACCESS_KEY" --env QA_LOG_LEVEL="$QA_LOG_LEVEL" --env QA_GITHUB_OAUTH_APP_ID="$QA_GITHUB_OAUTH_APP_ID" --env QA_EXPORT_TEST_METRICS="$QA_EXPORT_TEST_METRICS" --env QA_GOOGLE_CDN_LB="$QA_GOOGLE_CDN_LB" --env QA_1P_GITHUB_UUID="$QA_1P_GITHUB_UUID" --env QA_FACEBOOK_PASSWORD="$QA_FACEBOOK_PASSWORD" --env QA_GCS_CDN_BUCKET_NAME="$QA_GCS_CDN_BUCKET_NAME" --env QA_SCENARIO="$QA_SCENARIO" --env QA_GOOGLE_JSON_KEY="$QA_GOOGLE_JSON_KEY" --env QA_SUITES="$QA_SUITES" --env QA_RUN_ALL_E2E_LABEL="$QA_RUN_ALL_E2E_LABEL" --env QA_ALLURE_AWS_ACCESS_KEY_ID="$QA_ALLURE_AWS_ACCESS_KEY_ID" --env QA_AWS_S3_BUCKET_NAME="$QA_AWS_S3_BUCKET_NAME" --env QA_GENERATE_ALLURE_REPORT="$QA_GENERATE_ALLURE_REPORT" --env QA_CAN_TEST_PRAEFECT="$QA_CAN_TEST_PRAEFECT" --env QA_RUN_TESTS_ON_GDK="$QA_RUN_TESTS_ON_GDK" --env QA_INFLUXDB_TOKEN="$QA_INFLUXDB_TOKEN" --env QA_TEST_LICENSE_MODE="$QA_TEST_LICENSE_MODE" --env QA_TEST_SESSION_TOKEN="$QA_TEST_SESSION_TOKEN" --env QA_SLOW_CONNECTION_LATENCY_MS="$QA_SLOW_CONNECTION_LATENCY_MS" --env QA_RESULTS_ISSUE_PROJECT_TOKEN="$QA_RESULTS_ISSUE_PROJECT_TOKEN" --env QA_GITHUB_ACCESS_TOKEN="$QA_GITHUB_ACCESS_TOKEN" --env QA_FACEBOOK_USERNAME="$QA_FACEBOOK_USERNAME" --env QA_THIRD_PARTY_DOCKER_REPOSITORY="$QA_THIRD_PARTY_DOCKER_REPOSITORY" --env QA_SELENOID_BROWSER_VERSION="111.0" --env QA_VIDEO_RECORDER_IMAGE="presidenten/selenoid-manual-video-recorder" --env QA_VIDEO_RECORDER_VERSION="latest" --env QA_SELENOID_BROWSER_IMAGE="selenoid/chrome" --env GITLAB_QA_ADMIN_ACCESS_TOKEN="$GITLAB_QA_ADMIN_ACCESS_TOKEN" --env GITLAB_QA_PASSWORD_1="$GITLAB_QA_PASSWORD_1" --env GITLAB_QA_PASSWORD_2="$GITLAB_QA_PASSWORD_2" --env KNAPSACK_GENERATE_REPORT="$KNAPSACK_GENERATE_REPORT" --env CI="$CI" --env CI_JOB_NAME="$CI_JOB_NAME" --env CI_JOB_NAME_SLUG="$CI_JOB_NAME_SLUG" --env CI_JOB_URL="$CI_JOB_URL" --env CI_JOB_TOKEN="$CI_JOB_TOKEN" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env CI_PROJECT_NAME="$CI_PROJECT_NAME" --env CI_PROJECT_PATH="$CI_PROJECT_PATH" --env CI_PIPELINE_SOURCE="$CI_PIPELINE_SOURCE" --env CI_PIPELINE_URL="$CI_PIPELINE_URL" --env CI_PIPELINE_CREATED_AT="$CI_PIPELINE_CREATED_AT" --env GITLAB_CI="$GITLAB_CI" --env GITLAB_QA_USER_AGENT="$GITLAB_QA_USER_AGENT" --env GITLAB_INITIAL_ROOT_PASSWORD="5iveL!fe" --env COLORIZED_LOGS="$COLORIZED_LOGS" --env RELEASE="$RELEASE" --env RELEASE_REGISTRY_URL="registry.gitlab.com" --env RELEASE_REGISTRY_USERNAME="gitlab-ci-token" --env RELEASE_REGISTRY_PASSWORD="[MASKED]" --env SELENOID_DIRECTORY="/builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/selenoid" --env SCHEDULE_TYPE="$SCHEDULE_TYPE" --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-28-08-06-20-bd652493/gitlab-ee-qa-7eafdee1:/home/gitlab/qa/tmp:z --volume /var/run/docker.sock:/var/run/docker.sock:z --name gitlab-ee-qa-7eafdee1 registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:09612fea8bceb47ca2935844f4a8e1855a3f5898 Test::Instance::All http://gitlab-gitaly-cluster.test -- --format QA::Support::JsonFormatter --out tmp/rspec-4758982049.json --format RspecJunitFormatter --out tmp/rspec-4758982049.xml --format html --out tmp/rspec-gitaly-cluster.htm --color --format documentation --tag gitaly_cluster`
3302023-07-28 08:12:39 / CONF	:: 
331      ==> Base URL: http://gitlab-gitaly-cluster.test
332      ==> Browser: #<Selenium::WebDriver::Chrome::Driver:0x00007f12252a3990>
333      ==> Libraries: Chemlab::Vendor
334[Jul 28 2023 08:12:39 UTC (QA Tests)] INFO  -- Caching token for username: root, last six chars of token:sixwPA
335[Jul 28 2023 08:12:39 UTC (QA Tests)] INFO  -- Browser: chrome
336[Jul 28 2023 08:12:39 UTC (QA Tests)] INFO  -- Performing sanity check for environment!
337[Jul 28 2023 08:12:40 UTC (QA Tests)] INFO  -- Performing initial license fabrication!
338[Jul 28 2023 08:12:40 UTC (QA Tests)] INFO  -- Successfully added license key. Details:
339{:test_license_mode=>false, :license_key_length=>2136, :last_five_characters=>"cbiJ9", :plan=>"ultimate", :starts_at=>"2023-02-03", :user_limit=>10000}
340[Jul 28 2023 08:12:40 UTC (QA Tests)] INFO  -- ==> Built a QA::EE::Resource::License with id '1' via api in 0.26 seconds
341Run options:
342  include {:gitaly_cluster=>true}
343  exclude {:geo=>true, :requires_praefect=>true}
344Randomized with seed 45671
345[Jul 28 2023 08:12:41 UTC (QA Tests)] ERROR -- [Allure]: Failed to fetch flaky spec data for report: panic: sync: negative WaitGroup counter
346Systems
347  Gitaly distributed reads
348[Jul 28 2023 08:12:41 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly distributed reads reads from each node
349[Jul 28 2023 08:12:41 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
350[Jul 28 2023 08:12:41 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
351`
352[Jul 28 2023 08:12:41 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
353[Jul 28 2023 08:12:41 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
354[Jul 28 2023 08:12:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
355[Jul 28 2023 08:12:42 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
356[Jul 28 2023 08:12:42 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
357[Jul 28 2023 08:12:42 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
358[Jul 28 2023 08:12:43 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
359[Jul 28 2023 08:12:43 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
360[Jul 28 2023 08:12:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
361`
362[Jul 28 2023 08:12:44 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
363[Jul 28 2023 08:12:44 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
364[Jul 28 2023 08:12:45 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
365`
366[Jul 28 2023 08:12:45 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
367[Jul 28 2023 08:12:45 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
368[Jul 28 2023 08:12:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
369`
370[Jul 28 2023 08:12:46 UTC (QA Tests)] INFO  -- Retrieved cached token for username: root, last six chars of token:sixwPA
371[Jul 28 2023 08:12:46 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::PersonalAccessToken with name 'api-pat-root-quqg9g2x' via cache in 0.0 seconds
372[Jul 28 2023 08:12:47 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.94 seconds
373[Jul 28 2023 08:12:48 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907' via api in 1.89 seconds
374[Jul 28 2023 08:12:50 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907/gitaly_cluster-7effe0ac3c8a33be' via api in 4.16 seconds
375[Jul 28 2023 08:12:50 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
376`
377[Jul 28 2023 08:12:51 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[1]"'`
378[Jul 28 2023 08:13:19 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
379[Jul 28 2023 08:13:19 UTC (QA Tests)] INFO  -- Reading from the repository
380[Jul 28 2023 08:13:20 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
381[Jul 28 2023 08:13:25 UTC (QA Tests)] INFO  -- Reading from the repository
382[Jul 28 2023 08:13:25 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
383[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
384    reads from each node
385    when a node is unhealthy
386[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly distributed reads when a node is unhealthy does not read from the unhealthy node
387[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
388[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
389`
390[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
391[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
392[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
393[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
394[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
395[Jul 28 2023 08:13:26 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
396[Jul 28 2023 08:13:27 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
397[Jul 28 2023 08:13:27 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
398[Jul 28 2023 08:13:28 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
399`
400[Jul 28 2023 08:13:28 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
401[Jul 28 2023 08:13:28 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
402[Jul 28 2023 08:13:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
403`
404[Jul 28 2023 08:13:30 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
405[Jul 28 2023 08:13:30 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
406[Jul 28 2023 08:13:31 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
407`
408[Jul 28 2023 08:13:31 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.11 seconds
409[Jul 28 2023 08:13:31 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907' via api in 0.22 seconds
410[Jul 28 2023 08:13:33 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907/gitaly_cluster-ae69a45e10809fd5' via api in 2.43 seconds
411[Jul 28 2023 08:13:33 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
412`
413[Jul 28 2023 08:13:33 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[2]"'`
414[Jul 28 2023 08:14:02 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
415[Jul 28 2023 08:14:02 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
416[Jul 28 2023 08:14:02 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
417`
418[Jul 28 2023 08:14:03 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
419`
420[Jul 28 2023 08:14:04 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
421`
422[Jul 28 2023 08:14:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
423`
424[Jul 28 2023 08:14:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
425`
426[Jul 28 2023 08:14:08 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
427`
428[Jul 28 2023 08:14:09 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
429`
430[Jul 28 2023 08:14:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
431`
432[Jul 28 2023 08:14:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
433`
434[Jul 28 2023 08:14:11 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
435[Jul 28 2023 08:14:11 UTC (QA Tests)] INFO  -- Reading from the repository
436[Jul 28 2023 08:14:15 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
437[Jul 28 2023 08:14:17 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
438[Jul 28 2023 08:14:18 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
439[Jul 28 2023 08:14:19 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
440[Jul 28 2023 08:14:20 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
441[Jul 28 2023 08:14:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
442[Jul 28 2023 08:14:22 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
443[Jul 28 2023 08:14:23 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
444[Jul 28 2023 08:14:24 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
445[Jul 28 2023 08:14:24 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
446[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
447[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
448[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
449      does not read from the unhealthy node
450Systems
451  with Gitaly automatic failover and recovery
452[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
453[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
454`
455[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
456[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
457[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
458[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
459[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
460[Jul 28 2023 08:14:25 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
461[Jul 28 2023 08:14:26 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
462[Jul 28 2023 08:14:26 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
463[Jul 28 2023 08:14:27 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
464`
465[Jul 28 2023 08:14:28 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
466[Jul 28 2023 08:14:28 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
467[Jul 28 2023 08:14:29 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
468`
469[Jul 28 2023 08:14:29 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
470[Jul 28 2023 08:14:29 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
471[Jul 28 2023 08:14:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
472`
473[Jul 28 2023 08:14:30 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.06 seconds
474[Jul 28 2023 08:14:30 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907' via api in 0.22 seconds
475[Jul 28 2023 08:14:32 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907/gitaly_cluster-90ccb7d5f50abc07' via api in 2.33 seconds
476[Jul 28 2023 08:14:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
477`
478[Jul 28 2023 08:14:32 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`
479[Jul 28 2023 08:15:01 UTC (QA Tests)] INFO  -- Starting test: Systems with Gitaly automatic failover and recovery automatically fails over
480[Jul 28 2023 08:15:01 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
481[Jul 28 2023 08:15:01 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
482[Jul 28 2023 08:15:01 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
483`
484[Jul 28 2023 08:15:02 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
485`
486[Jul 28 2023 08:15:03 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
487`
488[Jul 28 2023 08:15:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
489`
490[Jul 28 2023 08:15:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
491`
492[Jul 28 2023 08:15:07 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
493`
494[Jul 28 2023 08:15:08 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
495`
496[Jul 28 2023 08:15:09 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
497`
498[Jul 28 2023 08:15:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
499`
500[Jul 28 2023 08:15:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
501`
502[Jul 28 2023 08:15:12 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
503[Jul 28 2023 08:15:12 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly3`
504[Jul 28 2023 08:15:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
505`
506[Jul 28 2023 08:15:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
507`
508[Jul 28 2023 08:15:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
509`
510[Jul 28 2023 08:15:15 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
511`
512[Jul 28 2023 08:15:16 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
513`
514[Jul 28 2023 08:15:18 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
515`
516[Jul 28 2023 08:15:19 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
517`
518[Jul 28 2023 08:15:20 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
519`
520[Jul 28 2023 08:15:21 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
521`
522[Jul 28 2023 08:15:23 UTC (QA Tests)] INFO  -- QA::Resource::Project - wait_for_push with commit message "first_added_commit_message to primary gitaly node"
523[Jul 28 2023 08:15:23 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.23 seconds
524[Jul 28 2023 08:15:23 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
525[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
526`
527[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
528[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
529[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
530[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
531[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
532[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly3`
533[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
534[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
535[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
536[Jul 28 2023 08:15:24 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
537[Jul 28 2023 08:15:25 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
538[Jul 28 2023 08:15:25 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
539[Jul 28 2023 08:15:26 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
540`
541[Jul 28 2023 08:15:27 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
542[Jul 28 2023 08:15:27 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
543[Jul 28 2023 08:15:28 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
544`
545[Jul 28 2023 08:15:28 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
546[Jul 28 2023 08:15:28 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
547[Jul 28 2023 08:15:29 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
548`
549[Jul 28 2023 08:15:29 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
550`
551[Jul 28 2023 08:15:29 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`
552[Jul 28 2023 08:15:58 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
553[Jul 28 2023 08:15:58 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly1`
554[Jul 28 2023 08:15:58 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
555`
556[Jul 28 2023 08:15:59 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
557`
558[Jul 28 2023 08:16:00 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
559`
560[Jul 28 2023 08:16:01 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
561`
562[Jul 28 2023 08:16:02 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
563`
564[Jul 28 2023 08:16:03 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
565`
566[Jul 28 2023 08:16:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
567`
568[Jul 28 2023 08:16:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
569`
570[Jul 28 2023 08:16:07 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
571`
572[Jul 28 2023 08:16:08 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
573`
574[Jul 28 2023 08:16:08 UTC (QA Tests)] INFO  -- Waiting for health check failure on gitaly1
575[Jul 28 2023 08:16:08 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
576`
577[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::Commit via api in 0.63 seconds
578    automatically fails over
579    when recovering from dataloss after failover
580[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- Starting test: Systems with Gitaly automatic failover and recovery when recovering from dataloss after failover automatically reconciles
581[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
582[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly1`
583[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
584[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
585[Jul 28 2023 08:16:09 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
586[Jul 28 2023 08:16:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
587`
588[Jul 28 2023 08:16:11 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`
589[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
590[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
591`
592[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
593[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
594[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
595[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
596[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
597[Jul 28 2023 08:16:39 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
598[Jul 28 2023 08:16:40 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
599[Jul 28 2023 08:16:40 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
600[Jul 28 2023 08:16:41 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
601`
602[Jul 28 2023 08:16:42 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
603[Jul 28 2023 08:16:42 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
604[Jul 28 2023 08:16:43 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
605`
606[Jul 28 2023 08:16:43 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
607[Jul 28 2023 08:16:43 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
608[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
609`
610      automatically reconciles
611Systems
612  Gitaly backend node recovery
613[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly backend node recovery recovers from dataloss
614[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
615[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
616`
617[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
618[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
619[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
620[Jul 28 2023 08:16:44 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
621[Jul 28 2023 08:16:45 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
622[Jul 28 2023 08:16:45 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
623[Jul 28 2023 08:16:46 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
624[Jul 28 2023 08:16:46 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
625[Jul 28 2023 08:16:47 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
626`
627[Jul 28 2023 08:16:47 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
628[Jul 28 2023 08:16:47 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
629[Jul 28 2023 08:16:48 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
630`
631[Jul 28 2023 08:16:48 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
632[Jul 28 2023 08:16:48 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
633[Jul 28 2023 08:16:49 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
634`
635[Jul 28 2023 08:16:49 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.11 seconds
636[Jul 28 2023 08:16:49 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907' via api in 0.22 seconds
637[Jul 28 2023 08:16:51 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907/gitaly_cluster-54b56a663ae4fcfa' via api in 2.3 seconds
638[Jul 28 2023 08:16:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
639`
640[Jul 28 2023 08:16:52 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
641[Jul 28 2023 08:17:20 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
642[Jul 28 2023 08:17:20 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly1`
643[Jul 28 2023 08:17:20 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
644`
645[Jul 28 2023 08:17:21 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
646`
647[Jul 28 2023 08:17:22 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
648`
649[Jul 28 2023 08:17:23 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
650`
651[Jul 28 2023 08:17:24 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
652`
653[Jul 28 2023 08:17:26 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
654`
655[Jul 28 2023 08:17:27 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
656`
657[Jul 28 2023 08:17:28 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
658`
659[Jul 28 2023 08:17:29 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
660`
661[Jul 28 2023 08:17:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
662`
663[Jul 28 2023 08:17:30 UTC (QA Tests)] INFO  -- Waiting for health check failure on gitaly1
664[Jul 28 2023 08:17:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
665`
666[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- QA::Resource::Project - wait_for_push with commit message "pushed after failover"
667[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 1.4 seconds
668[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select job from replication_queue
669where state = 'ready'
670  and job ->> 'change' = 'update'
671  and job ->> 'target_node_storage' = 'gitaly1';
672\""
673`
674[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
675[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly1`
676[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
677[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
678[Jul 28 2023 08:17:32 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
679[Jul 28 2023 08:17:33 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
680`
681[Jul 28 2023 08:17:34 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
682[Jul 28 2023 08:17:34 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
683[Jul 28 2023 08:17:35 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
684`
685[Jul 28 2023 08:17:35 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
686[Jul 28 2023 08:17:35 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
687[Jul 28 2023 08:17:36 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
688`
689[Jul 28 2023 08:17:36 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
690`
691[Jul 28 2023 08:17:36 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
692[Jul 28 2023 08:18:04 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
693[Jul 28 2023 08:18:04 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
694[Jul 28 2023 08:18:05 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
695`
696[Jul 28 2023 08:18:06 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
697`
698[Jul 28 2023 08:18:07 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
699`
700[Jul 28 2023 08:18:08 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
701`
702[Jul 28 2023 08:18:09 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
703`
704[Jul 28 2023 08:18:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
705`
706[Jul 28 2023 08:18:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
707`
708[Jul 28 2023 08:18:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
709`
710[Jul 28 2023 08:18:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
711`
712[Jul 28 2023 08:18:14 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
713[Jul 28 2023 08:18:14 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly3`
714[Jul 28 2023 08:18:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
715`
716[Jul 28 2023 08:18:15 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
717`
718[Jul 28 2023 08:18:16 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
719`
720[Jul 28 2023 08:18:18 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
721`
722[Jul 28 2023 08:18:19 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
723`
724[Jul 28 2023 08:18:20 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
725`
726[Jul 28 2023 08:18:21 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
727`
728[Jul 28 2023 08:18:22 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
729`
730[Jul 28 2023 08:18:23 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
731`
732[Jul 28 2023 08:18:24 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
733`
734[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
735[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
736`
737[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
738[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
739[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
740[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
741[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
742[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly3`
743[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
744[Jul 28 2023 08:18:25 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
745[Jul 28 2023 08:18:26 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
746[Jul 28 2023 08:18:26 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
747[Jul 28 2023 08:18:27 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
748[Jul 28 2023 08:18:27 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
749[Jul 28 2023 08:18:28 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
750`
751[Jul 28 2023 08:18:28 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
752[Jul 28 2023 08:18:28 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
753[Jul 28 2023 08:18:29 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
754`
755[Jul 28 2023 08:18:29 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
756[Jul 28 2023 08:18:29 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
757[Jul 28 2023 08:18:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
758`
759    recovers from dataloss
760Systems
761  Gitaly Cluster replication queue
762[Jul 28 2023 08:18:30 UTC (QA Tests)] INFO  -- Starting test: Systems Gitaly Cluster replication queue allows replication of different repository after interruption
763[Jul 28 2023 08:18:30 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
764[Jul 28 2023 08:18:30 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
765`
766[Jul 28 2023 08:18:30 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
767[Jul 28 2023 08:18:30 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
768[Jul 28 2023 08:18:31 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
769[Jul 28 2023 08:18:31 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
770[Jul 28 2023 08:18:31 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
771[Jul 28 2023 08:18:31 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
772[Jul 28 2023 08:18:32 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
773[Jul 28 2023 08:18:32 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
774[Jul 28 2023 08:18:33 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
775`
776[Jul 28 2023 08:18:33 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
777[Jul 28 2023 08:18:33 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
778[Jul 28 2023 08:18:34 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
779`
780[Jul 28 2023 08:18:34 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
781[Jul 28 2023 08:18:34 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
782[Jul 28 2023 08:18:35 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
783`
784[Jul 28 2023 08:18:35 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
785[Jul 28 2023 08:18:35 UTC (QA Tests)] INFO  -- Executing: `docker pause gitaly2`
786[Jul 28 2023 08:18:35 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
787`
788[Jul 28 2023 08:18:37 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
789`
790[Jul 28 2023 08:18:38 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
791`
792[Jul 28 2023 08:18:39 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
793`
794[Jul 28 2023 08:18:40 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
795`
796[Jul 28 2023 08:18:41 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
797`
798[Jul 28 2023 08:18:42 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
799`
800[Jul 28 2023 08:18:44 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
801`
802[Jul 28 2023 08:18:45 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
803`
804[Jul 28 2023 08:18:46 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
805`
806[Jul 28 2023 08:18:46 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.08 seconds
807[Jul 28 2023 08:18:46 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907' via api in 0.19 seconds
808[Jul 28 2023 08:18:47 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907/gitaly_cluster-bbe8f93295935844' via api in 1.14 seconds
809[Jul 28 2023 08:19:10 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
810[Jul 28 2023 08:19:10 UTC (QA Tests)] INFO  -- Executing: `docker unpause gitaly2`
811[Jul 28 2023 08:19:10 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
812[Jul 28 2023 08:19:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
813`
814[Jul 28 2023 08:19:10 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
815`
816[Jul 28 2023 08:19:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
817`
818[Jul 28 2023 08:19:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
819`
820[Jul 28 2023 08:19:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
821`
822[Jul 28 2023 08:19:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
823`
824[Jul 28 2023 08:19:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
825`
826[Jul 28 2023 08:19:11 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
827`
828[Jul 28 2023 08:19:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
829`
830[Jul 28 2023 08:19:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
831`
832[Jul 28 2023 08:19:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
833`
834[Jul 28 2023 08:19:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
835`
836[Jul 28 2023 08:19:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
837`
838[Jul 28 2023 08:19:12 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
839`
840[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
841`
842[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
843`
844[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
845`
846[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
847`
848[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
849`
850[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
851`
852[Jul 28 2023 08:19:13 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
853`
854[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
855`
856[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
857`
858[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
859`
860[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
861`
862[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
863`
864[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
865`
866[Jul 28 2023 08:19:14 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
867`
868[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
869`
870[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
871`
872[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
873[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker pause praefect`
874[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Setting jobs in replication queue to `in_progress` and acquiring locks
875[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"update replication_queue set state = 'in_progress';
876insert into replication_queue_job_lock (job_id, lock_id, triggered_at)
877  select id, rq.lock_id, created_at from replication_queue rq
878    left join replication_queue_job_lock rqjl on rq.id = rqjl.job_id
879    where state = 'in_progress' and rqjl.job_id is null;
880update replication_queue_lock set acquired = 't';
881\""
882`
883[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
884[Jul 28 2023 08:19:15 UTC (QA Tests)] INFO  -- Executing: `docker unpause praefect`
885[Jul 28 2023 08:19:16 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Running}} praefect`
886[Jul 28 2023 08:19:16 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
887[Jul 28 2023 08:19:16 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
888[Jul 28 2023 08:19:17 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.05 seconds
889[Jul 28 2023 08:19:17 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907' via api in 0.16 seconds
890[Jul 28 2023 08:19:18 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-07-28-08-12-46-1ec350ec5d086907/the_awesome_project-527a01f4a888e448' via api in 1.73 seconds
891[Jul 28 2023 08:19:18 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[6]"'`
892[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} postgres`
893[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
894`
895[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
896[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
897[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
898[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker inspect -f {{.State.Status}} praefect`
899[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Waiting for health check on praefect
900[Jul 28 2023 08:19:47 UTC (QA Tests)] INFO  -- Executing: `docker exec praefect gitlab-ctl status praefect`
901[Jul 28 2023 08:19:48 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly1
902[Jul 28 2023 08:19:48 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
903[Jul 28 2023 08:19:49 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
904`
905[Jul 28 2023 08:19:49 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly2
906[Jul 28 2023 08:19:49 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
907[Jul 28 2023 08:19:51 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
908`
909[Jul 28 2023 08:19:51 UTC (QA Tests)] INFO  -- Waiting for health check on gitaly3
910[Jul 28 2023 08:19:51 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
911[Jul 28 2023 08:19:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
912`
913[Jul 28 2023 08:19:52 UTC (QA Tests)] INFO  -- Clearing the replication queue
914[Jul 28 2023 08:19:52 UTC (QA Tests)] INFO  -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres   bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from replication_queue_job_lock;
915delete from replication_queue_lock;
916delete from replication_queue;
917\""
918`
919    allows replication of different repository after interruption
920Finished in 7 minutes 11 seconds (files took 3.17 seconds to load)
9216 examples, 0 failures
922Randomized with seed 45671
923[Jul 28 2023 08:19:53 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select n_live_tup, n_dead_tup, relname from pg_stat_all_tables order by n_live_tup DESC, n_dead_tup DESC;'"`
924[Jul 28 2023 08:19:53 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select * from pg_stat_user_tables;'"`
925[Jul 28 2023 08:19:53 UTC (Gitlab QA)] INFO  -- Shell command: `docker ps -f name=gitlab-gitaly-cluster`
926[Jul 28 2023 08:19:53 UTC (Gitlab QA)] INFO  -- Shell command: `docker rm -f gitlab-gitaly-cluster`
 927  Running after_script  
  00:25
928Running after script...
929$ if [ "$CI_JOB_STATUS" == "failed" ]; then # collapsed multi-line command
930$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
931$ echo -e "\e[0Ksection_start:`date +%s`:report_results_section[collapsed=true]\r\e[0KReport results"
 932  Report results  
  00:25
 954  Saving cache for successful job  
  00:01
955Not uploading cache qa-e2e-ruby-3.0-c62e56e5e86789fd869c3bcac450f355-16 due to policy
 956  Uploading artifacts for successful job  
  00:02
957Uploading artifacts...
958gitlab-qa-run-*: found 218 matching artifact files and directories 
959Uploading artifacts as "archive" to coordinator... 201 Created  id=4758982049 responseStatus=201 Created token=64_QuM-m
960Uploading artifacts...
961gitlab-qa-run-*/**/rspec-*.xml: found 1 matching artifact files and directories 
962Uploading artifacts as "junit" to coordinator... 201 Created  id=4758982049 responseStatus=201 Created token=64_QuM-m
963Uploading artifacts...
964/builds/gitlab-org/gitlab/suite_status.env: found 1 matching artifact files and directories 
965Uploading artifacts as "dotenv" to coordinator... 201 Created  id=4758982049 responseStatus=201 Created token=64_QuM-m
 966  Cleaning up project directory and file based variables  
  00:01
967Job succeeded
Tags: e2e
Job artifacts
The artifacts were removed 1 year ago
Commit 09612fea
Merge branch '413538-instance-setting-for-turning-off-the-ability-to-make-the-package-registry-public' into 'master'
Related jobs

decomposition-single-db 5/5
instance 4/5
mtls
_quarantine
cloud-activation
decomposition-multiple-db 1/5
decomposition-multiple-db 2/5
decomposition-multiple-db 3/5
decomposition-multiple-db 4/5
decomposition-multiple-db 5/5
decomposition-single-db 1/5
decomposition-single-db 2/5
decomposition-single-db 3/5
decomposition-single-db 4/5
elasticsearch
gitaly-cluster
gitlab-pages
group-saml
importers
instance 1/5
instance 2/5
instance 3/5
instance 5/5
instance-saml
integrations
jira
large-setup
ldap-no-server
ldap-no-tls
ldap-tls
mattermost
metrics
oauth
object-storage 1/2
object-storage 2/2
object-storage-aws 1/2
object-storage-aws 2/2
object-storage-gcs 1/2
object-storage-gcs 2/2
praefect 1/2
praefect 2/2
registry
registry-with-cdn
repository-storage
service-ping-disabled
smtp
update-major
update-minor