Skip to content
Snippets Groups Projects

gitaly-cluster

Passed Started by
🤖 GitLab Bot 🤖
@gitlab-bot
1Running with gitlab-runner 16.3.1 (f5dfa4d1)
2 on gitlab-org-qa-runner ss7o3c7p, system ID: s_6a7379124369
3 Resolving secrets 00:00
4 Preparing the "docker+machine" executor 01:16
5Using Docker executor with image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 ...
6Starting service docker:24.0.5-dind ...
7Pulling docker image docker:24.0.5-dind ...
8Using docker image sha256:7015f2c475d511a251955877c2862016a4042512ba625ed905e69202f87e1a21 for docker:24.0.5-dind with digest docker@sha256:3c6e4dca7a63c9a32a4e00da40461ce067f255987ccc9721cf18ffa087bcd1ef ...
9Waiting for services to be up and running (timeout 30 seconds)...
10Authenticating with credentials from job payload (GitLab Registry)
11Pulling docker image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 ...
12Using docker image sha256:844d06501af4b0f4f147629a9efbcda2c3f25bc9eed5e9fc173b6b9a54d08cf5 for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:a1b387c57b2646eb1c0dd64d7aa0a867e62996303e55af9e4837d56213ae4e6f ...
13 Preparing environment 00:00
14Running on runner-ss7o3c7p-project-278964-concurrent-0 via runner-ss7o3c7p-gitlab-org-qa-1701820268-c0ed0824-gcp...
15 Getting source from Git repository 00:21
16Fetching changes with git depth set to 20...
17Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/
18Created fresh repository.
19remote: Enumerating objects: 160361, done.
20remote: Counting objects: 100% (160361/160361), done.
21remote: Compressing objects: 100% (108335/108335), done.
22remote: Total 160361 (delta 73364), reused 105450 (delta 46010), pack-reused 0
23Receiving objects: 100% (160361/160361), 152.17 MiB | 42.55 MiB/s, done.
24Resolving deltas: 100% (73364/73364), done.
25From https://gitlab.com/gitlab-org/gitlab
26 * [new ref] refs/pipelines/1096679892 -> refs/pipelines/1096679892
27 * [new branch] master -> origin/master
28Checking out 807c767b as detached HEAD (ref is master)...
29Skipping Git submodules setup
30 Restoring cache 00:00
31Checking cache for qa-e2e-ruby-3.0-170e3eb0aa6d1cf29b552c828433d8e0-16...
32WARNING: file does not exist
33Failed to extract cache
34 Downloading artifacts 00:01
35Downloading artifacts for download-knapsack-report (5688711852)...
36Downloading artifacts from coordinator... ok host=storage.googleapis.com id=5688711852 responseStatus=200 OK token=64_GspgH
37Downloading artifacts for download-fast-quarantine-report (5688711853)...
38Downloading artifacts from coordinator... ok host=storage.googleapis.com id=5688711853 responseStatus=200 OK token=64_GspgH
39 Executing "step_script" stage of the job script 16:24
40Using docker image sha256:844d06501af4b0f4f147629a9efbcda2c3f25bc9eed5e9fc173b6b9a54d08cf5 for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-24.0.5 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:a1b387c57b2646eb1c0dd64d7aa0a867e62996303e55af9e4837d56213ae4e6f ...
41$ echo "SUITE_RAN=true" > $QA_SUITE_STATUS_ENV_FILE
42$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
43$ if [ "$RUN_WITH_BUNDLE" == "true" ]; then # collapsed multi-line command
44Fetching gem metadata from https://rubygems.org/.......
45Fetching rake 13.1.0
46Installing rake 13.1.0
47Fetching concurrent-ruby 1.2.2
48Fetching minitest 5.20.0
49Fetching erubi 1.12.0
50Fetching crass 1.0.6
51Fetching rack 2.2.3.1
52Fetching builder 3.2.4
53Fetching racc 1.7.1
54Fetching public_suffix 5.0.1
55Installing crass 1.0.6
56Installing erubi 1.12.0
57Installing builder 3.2.4
58Fetching http-accept 1.7.0
59Fetching unf_ext 0.0.8.2
60Installing racc 1.7.1 with native extensions
61Installing minitest 5.20.0
62Installing public_suffix 5.0.1
63Installing http-accept 1.7.0
64Installing rack 2.2.3.1
65Fetching mime-types-data 3.2023.0218.1
66Installing concurrent-ruby 1.2.2
67Fetching netrc 0.11.0
68Fetching rspec-support 3.12.0
69Installing netrc 0.11.0
70Fetching diff-lcs 1.3
71Installing rspec-support 3.12.0
72Installing mime-types-data 3.2023.0218.1
73Installing unf_ext 0.0.8.2 with native extensions
74Fetching require_all 3.0.0
75Installing diff-lcs 1.3
76Installing require_all 3.0.0
77Fetching systemu 2.6.5
78Fetching sexp_processor 4.17.0
79Fetching sync 0.5.0
80Installing systemu 2.6.5
81Fetching ast 2.4.2
82Fetching debug_inspector 1.1.0
83Installing sync 0.5.0
84Fetching byebug 11.1.3
85Installing sexp_processor 4.17.0
86Fetching matrix 0.4.2
87Installing debug_inspector 1.1.0 with native extensions
88Installing ast 2.4.2
89Fetching mini_mime 1.1.0
90Fetching regexp_parser 2.1.1
91Installing byebug 11.1.3 with native extensions
92Installing matrix 0.4.2
93Fetching colorize 0.8.1
94Installing mini_mime 1.1.0
95Fetching rubyzip 2.3.2
96Installing regexp_parser 2.1.1
97Fetching websocket 1.2.10
98Installing colorize 0.8.1
99Installing rubyzip 2.3.2
100Fetching coderay 1.1.2
101Installing websocket 1.2.10
102Installing coderay 1.1.2
103Fetching multi_xml 0.6.0
104Fetching unicode-display_width 2.4.2
105Installing multi_xml 0.6.0
106Fetching zeitwerk 2.6.12
107Fetching declarative 0.0.20
108Installing unicode-display_width 2.4.2
109Fetching excon 0.92.4
110Fetching faraday-net_http 3.0.0
111Fetching ruby2_keywords 0.0.5
112Installing declarative 0.0.20
113Installing zeitwerk 2.6.12
114Fetching ffi 1.15.5
115Fetching formatador 0.3.0
116Installing ruby2_keywords 0.0.5
117Fetching multi_json 1.15.0
118Installing faraday-net_http 3.0.0
119Fetching jwt 2.5.0
120Installing formatador 0.3.0
121Installing excon 0.92.4
122Fetching memoist 0.16.2
123Installing multi_json 1.15.0
124Fetching os 1.1.4
125Installing jwt 2.5.0
126Installing memoist 0.16.2
127Fetching httpclient 2.8.3
128Installing os 1.1.4
129Fetching trailblazer-option 0.1.2
130Fetching uber 0.1.0
131Installing trailblazer-option 0.1.2
132Fetching retriable 3.1.2
133Installing uber 0.1.0
134Fetching webrick 1.7.0
135Fetching http-form_data 2.3.0
136Installing retriable 3.1.2
137Fetching parallel 1.23.0
138Installing httpclient 2.8.3
139Fetching rainbow 3.1.1
140Installing http-form_data 2.3.0
141Installing parallel 1.23.0
142Installing webrick 1.7.0
143Fetching table_print 1.5.7
144Installing ffi 1.15.5 with native extensions
145Installing rainbow 3.1.1
146Fetching influxdb-client 2.9.0
147Installing table_print 1.5.7
148Fetching knapsack 4.0.0
149Fetching method_source 1.0.0
150Fetching rotp 6.3.0
151Installing influxdb-client 2.9.0
152Fetching ruby-debug-ide 0.7.3
153Installing rotp 6.3.0
154Installing method_source 1.0.0
155Fetching slack-notifier 2.4.0
156Fetching warning 1.3.0
157Fetching addressable 2.8.1
158Installing slack-notifier 2.4.0
159Installing ruby-debug-ide 0.7.3 with native extensions
160Installing warning 1.3.0
161Fetching rspec-core 3.12.0
162Fetching rack-test 1.1.0
163Fetching mime-types 3.4.1
164Installing knapsack 4.0.0
165Installing addressable 2.8.1
166Installing rack-test 1.1.0
167Fetching rspec-expectations 3.12.0
168Installing mime-types 3.4.1
169Installing rspec-core 3.12.0
170Fetching rspec-mocks 3.12.0
171Fetching macaddr 1.7.2
172Installing rspec-expectations 3.12.0
173Installing macaddr 1.7.2
174Installing rspec-mocks 3.12.0
175Fetching tins 1.32.1
176Fetching parser 3.2.2.1
177Fetching ruby_parser 3.20.3
178Fetching i18n 1.12.0
179Fetching tzinfo 2.0.6
180Installing tins 1.32.1
181Installing i18n 1.12.0
182Installing tzinfo 2.0.6
183Fetching binding_of_caller 1.0.0
184Fetching selenium-webdriver 4.15.0
185Installing binding_of_caller 1.0.0
186Fetching httparty 0.21.0
187Fetching terminal-table 3.0.2
188Installing ruby_parser 3.20.3
189Installing parser 3.2.2.1
190Installing terminal-table 3.0.2
191Installing httparty 0.21.0
192Fetching faraday 2.5.2
193Fetching nokogiri 1.15.5 (x86_64-linux)
194Fetching representable 3.2.0
195Installing faraday 2.5.2
196Installing representable 3.2.0
197Fetching parallel_tests 4.3.0
198Fetching unf 0.1.4
199Installing parallel_tests 4.3.0
200Fetching pry 0.14.1
201Fetching launchy 2.5.0
202Installing unf 0.1.4
203Fetching fog-core 2.1.0
204Fetching uuid 2.3.9
205Installing pry 0.14.1
206Installing launchy 2.5.0
207Installing fog-core 2.1.0
208Installing uuid 2.3.9
209Fetching rspec-retry 0.6.2
210Fetching rspec_junit_formatter 0.6.0
211Installing rspec_junit_formatter 0.6.0
212Installing rspec-retry 0.6.2
213Fetching rspec 3.12.0
214Fetching faker 3.2.2
215Installing rspec 3.12.0
216Fetching activesupport 7.0.8
217Fetching gitlab 4.19.0
218Fetching protocol 2.0.0
219Installing nokogiri 1.15.5 (x86_64-linux)
220Installing protocol 2.0.0
221Installing gitlab 4.19.0
222Installing activesupport 7.0.8
223Fetching faraday-retry 2.2.0
224Installing faraday-retry 2.2.0
225Fetching signet 0.17.0
226Fetching google-cloud-env 1.6.0
227Installing google-cloud-env 1.6.0
228Installing signet 0.17.0
229Fetching sawyer 0.9.2
230Fetching unparser 0.6.5
231Installing faker 3.2.2
232Installing sawyer 0.9.2
233Installing unparser 0.6.5
234Fetching domain_name 0.5.20190701
235Installing domain_name 0.5.20190701
236Fetching allure-ruby-commons 2.23.0
237Fetching fog-json 1.2.0
238Fetching pry-byebug 3.10.1
239Installing allure-ruby-commons 2.23.0
240Installing fog-json 1.2.0
241Installing pry-byebug 3.10.1
242Fetching mize 0.4.1
243Fetching confiner 0.4.0
244Fetching googleauth 1.2.0
245Installing mize 0.4.1
246Installing confiner 0.4.0
247Installing selenium-webdriver 4.15.0
248Fetching octokit 8.0.0
249Installing googleauth 1.2.0
250Fetching proc_to_ast 0.1.0
251Fetching deprecation_toolkit 2.0.4
252Installing proc_to_ast 0.1.0
253Installing octokit 8.0.0
254Installing deprecation_toolkit 2.0.4
255Fetching factory_bot 6.3.0
256Fetching http-cookie 1.0.5
257Installing factory_bot 6.3.0
258Installing http-cookie 1.0.5
259Fetching allure-rspec 2.23.0
260Fetching amatch 0.4.1
261Installing allure-rspec 2.23.0
262Fetching google-apis-core 0.9.0
263Fetching rspec-parameterized-core 1.0.0
264Fetching rest-client 2.1.0
265Installing amatch 0.4.1 with native extensions
266Installing rspec-parameterized-core 1.0.0
267Installing google-apis-core 0.9.0
268Fetching rails-dom-testing 2.0.3
269Fetching loofah 2.21.3
270Installing rest-client 2.1.0
271Installing rails-dom-testing 2.0.3
272Fetching xpath 3.2.0
273Installing loofah 2.21.3
274Installing xpath 3.2.0
275Fetching fog-xml 0.1.4
276Fetching rspec-parameterized-table_syntax 1.0.0
277Fetching google-apis-compute_v1 0.51.0
278Installing fog-xml 0.1.4
279Installing rspec-parameterized-table_syntax 1.0.0
280Fetching google-apis-dns_v1 0.27.0
281Fetching google-apis-iamcredentials_v1 0.14.0
282Fetching google-apis-monitoring_v3 0.33.0
283Installing google-apis-iamcredentials_v1 0.14.0
284Fetching google-apis-pubsub_v1 0.28.0
285Installing google-apis-dns_v1 0.27.0
286Installing google-apis-compute_v1 0.51.0
287Installing google-apis-monitoring_v3 0.33.0
288Fetching google-apis-sqladmin_v1beta4 0.36.0
289Installing google-apis-pubsub_v1 0.28.0
290Fetching google-apis-storage_v1 0.19.0
291Fetching rails-html-sanitizer 1.5.0
292Fetching capybara 3.39.2
293Installing google-apis-sqladmin_v1beta4 0.36.0
294Installing google-apis-storage_v1 0.19.0
295Fetching rspec-parameterized 1.0.0
296Fetching airborne 0.3.7
297Installing rails-html-sanitizer 1.5.0
298Fetching fog-google 1.19.0
299Installing rspec-parameterized 1.0.0
300Installing airborne 0.3.7
301Installing capybara 3.39.2
302Fetching actionview 7.0.8
303Installing fog-google 1.19.0
304Installing actionview 7.0.8
305Fetching watir 7.1.0
306Installing watir 7.1.0
307Fetching capybara-screenshot 1.0.26
308Installing capybara-screenshot 1.0.26
309Fetching chemlab 0.11.1
310Installing chemlab 0.11.1
311Fetching chemlab-library-www-gitlab-com 0.1.1
312Installing chemlab-library-www-gitlab-com 0.1.1
313Fetching ffi-compiler 1.0.1
314Installing ffi-compiler 1.0.1
315Fetching llhttp-ffi 0.4.0
316Installing llhttp-ffi 0.4.0 with native extensions
317Fetching http 5.1.1
318Installing http 5.1.1
319Fetching gitlab-qa 12.5.1
320Fetching gitlab_quality-test_tooling 1.8.1
321Installing gitlab_quality-test_tooling 1.8.1
322Installing gitlab-qa 12.5.1
323Bundle complete! 38 Gemfile dependencies, 142 gems now installed.
324Use `bundle info [gemname]` to see where a bundled gem is installed.
325Post-install message from rubyzip:
326RubyZip 3.0 is coming!
327**********************
328The public API of some Rubyzip classes has been modernized to use named
329parameters for optional arguments. Please check your usage of the
330following classes:
331 * `Zip::File`
332 * `Zip::Entry`
333 * `Zip::InputStream`
334 * `Zip::OutputStream`
335Please ensure that your Gemfiles and .gemspecs are suitably restrictive
336to avoid an unexpected breakage when 3.0 is released (e.g. ~> 2.3.0).
337See https://github.com/rubyzip/rubyzip for details. The Changelog also
338lists other enhancements and bugfixes that have been implemented since
339version 2.3.0.
340Post-install message from httparty:
341When you HTTParty, you must party hard!
342$ export RSPEC_FAST_QUARANTINE_PATH="/home/gitlab/qa/${RSPEC_FAST_QUARANTINE_PATH}"
343$ export RSPEC_FAST_QUARANTINE_PATH="/home/gitlab/qa/${RSPEC_FAST_QUARANTINE_PATH}"
344$ export RSPEC_SKIPPED_TESTS_REPORT_PATH="/home/gitlab/qa/rspec/skipped_tests-${CI_JOB_ID}.txt"
345$ export QA_COMMAND="$BUNDLE_PREFIX gitlab-qa ${QA_SCENARIO:=Test::Instance::Image} $RELEASE $GITLAB_QA_OPTS -- $QA_TESTS $QA_RSPEC_TAGS $RSPEC_REPORT_OPTS"
346$ echo "Running - '$QA_COMMAND'"
347Running - 'bundle exec gitlab-qa Test::Integration::GitalyCluster registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5 -- --format QA::Support::JsonFormatter --out tmp/rspec-5688711924.json --format RspecJunitFormatter --out tmp/rspec-5688711924.xml --format html --out tmp/rspec-$(echo gitaly-cluster | sed 's/[ /]/_/g').htm --color --format documentation'
348$ eval "$QA_COMMAND"
349[Dec 05 2023 23:58:12 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q postgres:11`
350[Dec 05 2023 23:58:17 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^postgres$`
351[Dec 05 2023 23:58:17 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
352[Dec 05 2023 23:58:17 UTC (Gitlab QA)] INFO -- Shell command: `docker network create test`
353[Dec 05 2023 23:58:17 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name postgres --net test --env POSTGRES_PASSWORD="SQL_PASSWORD" postgres:11`
354[Dec 05 2023 23:58:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`
355[Dec 05 2023 23:58:21 UTC (Gitlab QA)] ERROR -- Shell command output:
356psql: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
357 Is the server running locally and accepting connections on that socket?
358[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`
359[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Shell command: `docker exec postgres bash -c "psql -U postgres -d template1 -c \"CREATE DATABASE praefect_production OWNER postgres\""`
360[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Starting Gitaly Cluster
361[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^alpine-d3e88d06$`
362[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
363[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name alpine-d3e88d06 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z alpine/curl:latest tail -f /dev/null`
364[Dec 05 2023 23:58:26 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
365[Dec 05 2023 23:58:34 UTC (Gitlab QA)] INFO -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.5.1/tls_certificates/authority/. alpine-d3e88d06:/etc/gitlab/trusted-certs`
366[Dec 05 2023 23:58:34 UTC (Gitlab QA)] INFO -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.5.1/tls_certificates/gitlab/. alpine-d3e88d06:/etc/gitlab/ssl`
367[Dec 05 2023 23:58:34 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=alpine-d3e88d06`
368[Dec 05 2023 23:58:34 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f alpine-d3e88d06`
369[Dec 05 2023 23:58:37 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
370[Dec 05 2023 23:58:38 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
371[Dec 05 2023 23:59:17 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly1$`
372[Dec 05 2023 23:59:17 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitlab-gitaly-cluster$`
373[Dec 05 2023 23:59:17 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
374[Dec 05 2023 23:59:17 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
375[Dec 05 2023 23:59:17 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitlab-gitaly-cluster --net test --hostname gitlab-gitaly-cluster.test -p 80 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/gitlab-gitaly-cluster/logs:/var/log/gitlab:Z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
376[Dec 05 2023 23:59:17 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly1 --net test --hostname gitaly1.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/gitaly1/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
377[Dec 05 2023 23:59:29 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "echo \"postgresql['enable'] = false;
378redis['enable'] = false;
379nginx['enable'] = false;
380puma['enable'] = false;
381sidekiq['enable'] = false;
382gitlab_workhorse['enable'] = false;
383gitlab_rails['rake_cache_clear'] = false;
384gitlab_rails['auto_migrate'] = false;
385gitlab_exporter['enable'] = false;
386gitlab_kas['enable'] = false;
387praefect['enable'] = false;
388prometheus['enable'] = true;
389gitaly['enable'] = true;
390gitaly['configuration'] = {
391 'listen_addr': '0.0.0.0:8075',
392 'prometheus_listen_addr': '0.0.0.0:9236',
393 'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
394}
395gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
396gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
397git_data_dirs({
398 'gitaly1' => {
399 'path' => '/var/opt/gitlab/git-data'
400 },
401 'gitaly2' => {
402 'path' => '/var/opt/gitlab/git-data'
403 },
404 'gitaly3' => {
405 'path' => '/var/opt/gitlab/git-data'
406 }
407});
408\" > /etc/gitlab/gitlab.rb;"`
409[Dec 05 2023 23:59:29 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly1`
410[Dec 05 2023 23:59:29 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo \"# Generated by GitLab QA Omnibus Configurator at 2023-12-05 23:58:12
411gitlab_rails['gitlab_default_theme'] = 10 # Light Red Theme
412gitlab_rails['gitlab_disable_animations'] = true # Disable animations
413gitlab_rails['application_settings_cache_seconds'] = 0 # Settings cache expiry
414gitlab_rails['initial_root_password'] = '5iveL!fe' # Initial root password
415external_url 'http://gitlab-gitaly-cluster.test';
416git_data_dirs({
417 'default' => {
418 'gitaly_address' => 'tcp://praefect.test:2305',
419 'gitaly_token' => 'PRAEFECT_EXTERNAL_TOKEN'
420 }
421});
422gitaly['enable'] = false;
423gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
424gitlab_rails['env'] = {'GITLAB_LICENSE_MODE'=>'test', 'CUSTOMER_PORTAL_URL'=>'https://customers.staging.gitlab.com', 'GITLAB_QA_USER_AGENT'=>'*****', 'GITLAB_ALLOW_SEPARATE_CI_DATABASE'=>'true'}
425prometheus['scrape_configs'] = [
426 {
427 'job_name' => 'praefect',
428 'static_configs' => [
429 'targets' => [
430 'praefect.test:9652'
431 ]
432 ]
433 },
434 {
435 'job_name' => 'praefect-gitaly',
436 'static_configs' => [
437 'targets' => [
438 'gitaly1.test:9236',
439 'gitaly2.test:9236',
440 'gitaly3.test:9236'
441 ]
442 ]
443 }]\" > /etc/gitlab/gitlab.rb;"`
444[Dec 05 2023 23:59:29 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitlab-gitaly-cluster`
445[Dec 06 2023 00:00:13 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
446[Dec 06 2023 00:00:13 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "echo '#!/usr/bin/env bash
447if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
448 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
449 exit 1
450fi
451' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
452[Dec 06 2023 00:00:13 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
453[Dec 06 2023 00:00:13 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
454[Dec 06 2023 00:00:14 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
455[Dec 06 2023 00:00:14 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly2$`
456[Dec 06 2023 00:00:14 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
457[Dec 06 2023 00:00:14 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly2 --net test --hostname gitaly2.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/gitaly2/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
458[Dec 06 2023 00:00:15 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "echo \"postgresql['enable'] = false;
459redis['enable'] = false;
460nginx['enable'] = false;
461puma['enable'] = false;
462sidekiq['enable'] = false;
463gitlab_workhorse['enable'] = false;
464gitlab_rails['rake_cache_clear'] = false;
465gitlab_rails['auto_migrate'] = false;
466gitlab_exporter['enable'] = false;
467gitlab_kas['enable'] = false;
468praefect['enable'] = false;
469prometheus['enable'] = true;
470gitaly['enable'] = true;
471gitaly['configuration'] = {
472 'listen_addr': '0.0.0.0:8075',
473 'prometheus_listen_addr': '0.0.0.0:9236',
474 'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
475}
476gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
477gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
478git_data_dirs({
479 'gitaly1' => {
480 'path' => '/var/opt/gitlab/git-data'
481 },
482 'gitaly2' => {
483 'path' => '/var/opt/gitlab/git-data'
484 },
485 'gitaly3' => {
486 'path' => '/var/opt/gitlab/git-data'
487 }
488});
489\" > /etc/gitlab/gitlab.rb;"`
490[Dec 06 2023 00:00:15 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly2`
491[Dec 06 2023 00:00:59 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
492[Dec 06 2023 00:00:59 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "echo '#!/usr/bin/env bash
493if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
494 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
495 exit 1
496fi
497' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
498[Dec 06 2023 00:00:59 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
499[Dec 06 2023 00:00:59 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
500[Dec 06 2023 00:00:59 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
501[Dec 06 2023 00:01:00 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly3$`
502[Dec 06 2023 00:01:00 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
503[Dec 06 2023 00:01:00 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly3 --net test --hostname gitaly3.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/gitaly3/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
504[Dec 06 2023 00:01:01 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "echo \"postgresql['enable'] = false;
505redis['enable'] = false;
506nginx['enable'] = false;
507puma['enable'] = false;
508sidekiq['enable'] = false;
509gitlab_workhorse['enable'] = false;
510gitlab_rails['rake_cache_clear'] = false;
511gitlab_rails['auto_migrate'] = false;
512gitlab_exporter['enable'] = false;
513gitlab_kas['enable'] = false;
514praefect['enable'] = false;
515prometheus['enable'] = true;
516gitaly['enable'] = true;
517gitaly['configuration'] = {
518 'listen_addr': '0.0.0.0:8075',
519 'prometheus_listen_addr': '0.0.0.0:9236',
520 'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
521}
522gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
523gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
524git_data_dirs({
525 'gitaly1' => {
526 'path' => '/var/opt/gitlab/git-data'
527 },
528 'gitaly2' => {
529 'path' => '/var/opt/gitlab/git-data'
530 },
531 'gitaly3' => {
532 'path' => '/var/opt/gitlab/git-data'
533 }
534});
535\" > /etc/gitlab/gitlab.rb;"`
536[Dec 06 2023 00:01:01 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly3`
537[Dec 06 2023 00:01:46 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
538[Dec 06 2023 00:01:46 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "echo '#!/usr/bin/env bash
539if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
540 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
541 exit 1
542fi
543' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
544[Dec 06 2023 00:01:46 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
545[Dec 06 2023 00:01:47 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
546[Dec 06 2023 00:01:47 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
547[Dec 06 2023 00:01:47 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^praefect$`
548[Dec 06 2023 00:01:47 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
549[Dec 06 2023 00:01:47 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name praefect --net test --hostname praefect.test -p 2305 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/praefect/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:807c767b70d28221a50c9acb95390fece870a5c5`
550[Dec 06 2023 00:01:48 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "echo \"postgresql['enable'] = false;
551redis['enable'] = false;
552nginx['enable'] = false;
553puma['enable'] = false;
554sidekiq['enable'] = false;
555gitlab_workhorse['enable'] = false;
556gitlab_rails['rake_cache_clear'] = false;
557gitlab_rails['auto_migrate'] = false;
558gitlab_exporter['enable'] = false;
559gitlab_kas['enable'] = false;
560gitaly['enable'] = false;
561prometheus['enable'] = true;
562praefect['enable'] = true;
563praefect['configuration'] = {
564 listen_addr: '0.0.0.0:2305',
565 prometheus_listen_addr: '0.0.0.0:9652',
566 auth: {
567 token: 'PRAEFECT_EXTERNAL_TOKEN'
568 },
569 reconciliation: {
570 scheduling_interval: '10s'
571 },
572 database: {
573 host: 'postgres.test',
574 port: 5432,
575 user: 'postgres',
576 password: 'SQL_PASSWORD',
577 dbname: 'praefect_production',
578 sslmode: 'disable'
579 },
580 virtual_storage: [
581 {
582 name: 'default',
583 node: [
584 {
585 'storage': 'gitaly1',
586 'address': 'tcp://gitaly1.test:8075',
587 'token': 'PRAEFECT_INTERNAL_TOKEN'
588 },
589 {
590 'storage': 'gitaly2',
591 'address': 'tcp://gitaly2.test:8075',
592 'token': 'PRAEFECT_INTERNAL_TOKEN'
593 },
594 {
595 'storage': 'gitaly3',
596 'address': 'tcp://gitaly3.test:8075',
597 'token': 'PRAEFECT_INTERNAL_TOKEN'
598 }
599 ],
600 }
601 ]
602}
603\" > /etc/gitlab/gitlab.rb;"`
604[Dec 06 2023 00:01:48 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false praefect`
605[Dec 06 2023 00:02:34 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "praefect -config /var/opt/gitlab/praefect/cluster_config.toml check || true"`
606[Dec 06 2023 00:02:34 UTC (Gitlab QA)] INFO -- Shell command: `docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' praefect`
607[Dec 06 2023 00:02:34 UTC (Gitlab QA)] INFO -- Gitaly Cluster Ready
608[Dec 06 2023 00:02:56 UTC (Gitlab QA)] INFO -- Waiting for GitLab to become healthy ...
609[Dec 06 2023 00:02:56 UTC (Gitlab QA)] INFO -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-gitaly-cluster`
610[Dec 06 2023 00:02:56 UTC (Gitlab QA)] INFO -- Shell command: `docker port gitlab-gitaly-cluster 80/tcp`
611[Dec 06 2023 00:02:56 UTC (Gitlab QA)] INFO -- -> GitLab is available at `http://docker:32769`!
612[Dec 06 2023 00:02:56 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
613[Dec 06 2023 00:02:56 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo '#!/usr/bin/env bash
614if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
615 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
616 exit 1
617fi
618' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
619[Dec 06 2023 00:02:57 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
620[Dec 06 2023 00:02:57 UTC (Gitlab QA)] INFO -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-12.5.1/support/data gitlab-gitaly-cluster:/tmp/data-seeds`
621[Dec 06 2023 00:02:57 UTC (Gitlab QA)] INFO -- Running exec_commands...
622[Dec 06 2023 00:02:57 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-rails runner /tmp/data-seeds/admin_access_token_seed.rb"`
623[Dec 06 2023 00:03:46 UTC (Gitlab QA)] INFO -- Running Gitaly Cluster specs!
624[Dec 06 2023 00:03:46 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
625[Dec 06 2023 00:03:46 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:807c767b70d28221a50c9acb95390fece870a5c5`
626[Dec 06 2023 00:04:53 UTC (Gitlab QA)] INFO -- Running test suite `Test::Instance::All` for gitlab-ee
627[Dec 06 2023 00:04:53 UTC (Gitlab QA)] INFO -- Shell command: `docker run -t --rm --net=test --env QA_AWS_S3_REGION="$QA_AWS_S3_REGION" --env QA_FACEBOOK_OAUTH_APP_ID="$QA_FACEBOOK_OAUTH_APP_ID" --env QA_GOOGLE_CDN_JSON_KEY="$QA_GOOGLE_CDN_JSON_KEY" --env QA_SUITE_STATUS_ENV_FILE="$QA_SUITE_STATUS_ENV_FILE" --env QA_THIRD_PARTY_DOCKER_REGISTRY="$QA_THIRD_PARTY_DOCKER_REGISTRY" --env QA_RSPEC_REPORT_PATH="$QA_RSPEC_REPORT_PATH" --env QA_SAVE_TEST_METRICS="$QA_SAVE_TEST_METRICS" --env QA_GOOGLE_CDN_SIGNURL_KEY_NAME="$QA_GOOGLE_CDN_SIGNURL_KEY_NAME" --env QA_1P_PASSWORD="$QA_1P_PASSWORD" --env QA_REMOTE_GRID_USERNAME="$QA_REMOTE_GRID_USERNAME" --env QA_GOOGLE_CDN_SIGNURL_KEY="$QA_GOOGLE_CDN_SIGNURL_KEY" --env QA_TEST_CASE_PROJECT_TOKEN="$QA_TEST_CASE_PROJECT_TOKEN" --env QA_GITHUB_USERNAME="$QA_GITHUB_USERNAME" --env QA_SLOW_CONNECTION_THROUGHPUT_KBPS="$QA_SLOW_CONNECTION_THROUGHPUT_KBPS" --env QA_GITHUB_PASSWORD="$QA_GITHUB_PASSWORD" --env QA_ARTIFACTS_DIR="$QA_ARTIFACTS_DIR" --env QA_GITLAB_TLS_CERTIFICATE="$QA_GITLAB_TLS_CERTIFICATE" --env QA_RUN_TYPE="$QA_RUN_TYPE" --env QA_JIRA_ADMIN_PASSWORD="$QA_JIRA_ADMIN_PASSWORD" --env QA_GITHUB_OAUTH_APP_SECRET="$QA_GITHUB_OAUTH_APP_SECRET" --env QA_PATH="$QA_PATH" --env QA_JIRA_ADMIN_USERNAME="$QA_JIRA_ADMIN_USERNAME" --env QA_ADMIN_ACCESS_TOKEN="$QA_ADMIN_ACCESS_TOKEN" --env QA_COMMAND="$QA_COMMAND" --env QA_EE_LICENSE="$QA_EE_LICENSE" --env QA_1P_EMAIL="$QA_1P_EMAIL" --env QA_EE_ACTIVATION_CODE="$QA_EE_ACTIVATION_CODE" --env QA_GOOGLE_CLIENT_EMAIL="$QA_GOOGLE_CLIENT_EMAIL" --env QA_ALLURE_AWS_SECRET_ACCESS_KEY="$QA_ALLURE_AWS_SECRET_ACCESS_KEY" --env QA_GCS_BUCKET_NAME="$QA_GCS_BUCKET_NAME" --env QA_FACEBOOK_OAUTH_APP_SECRET="$QA_FACEBOOK_OAUTH_APP_SECRET" --env QA_GOOGLE_PROJECT="$QA_GOOGLE_PROJECT" --env QA_KNAPSACK_REPORT_GCS_CREDENTIALS="$QA_KNAPSACK_REPORT_GCS_CREDENTIALS" --env QA_AWS_S3_KEY_ID="$QA_AWS_S3_KEY_ID" --env QA_RUN_ALL_TESTS="$QA_RUN_ALL_TESTS" --env QA_GITLAB_CI_TOKEN="$QA_GITLAB_CI_TOKEN" --env QA_REMOTE_GRID_ACCESS_KEY="$QA_REMOTE_GRID_ACCESS_KEY" --env QA_FRAMEWORK_CHANGES="$QA_FRAMEWORK_CHANGES" --env QA_INTERCEPT_REQUESTS="$QA_INTERCEPT_REQUESTS" --env QA_INFLUXDB_URL="$QA_INFLUXDB_URL" --env QA_THIRD_PARTY_DOCKER_USER="$QA_THIRD_PARTY_DOCKER_USER" --env QA_THIRD_PARTY_DOCKER_PASSWORD="$QA_THIRD_PARTY_DOCKER_PASSWORD" --env QA_1P_SECRET="$QA_1P_SECRET" --env QA_AWS_S3_ACCESS_KEY="$QA_AWS_S3_ACCESS_KEY" --env QA_LOG_LEVEL="$QA_LOG_LEVEL" --env QA_GITHUB_OAUTH_APP_ID="$QA_GITHUB_OAUTH_APP_ID" --env QA_EXPORT_TEST_METRICS="$QA_EXPORT_TEST_METRICS" --env QA_GOOGLE_CDN_LB="$QA_GOOGLE_CDN_LB" --env QA_1P_GITHUB_UUID="$QA_1P_GITHUB_UUID" --env QA_FACEBOOK_PASSWORD="$QA_FACEBOOK_PASSWORD" --env QA_GCS_CDN_BUCKET_NAME="$QA_GCS_CDN_BUCKET_NAME" --env QA_SCENARIO="$QA_SCENARIO" --env QA_GOOGLE_JSON_KEY="$QA_GOOGLE_JSON_KEY" --env QA_SUITES="$QA_SUITES" --env QA_RUN_ALL_E2E_LABEL="$QA_RUN_ALL_E2E_LABEL" --env QA_ALLURE_AWS_ACCESS_KEY_ID="$QA_ALLURE_AWS_ACCESS_KEY_ID" --env QA_AWS_S3_BUCKET_NAME="$QA_AWS_S3_BUCKET_NAME" --env QA_GENERATE_ALLURE_REPORT="$QA_GENERATE_ALLURE_REPORT" --env QA_CAN_TEST_PRAEFECT="$QA_CAN_TEST_PRAEFECT" --env QA_RUN_TESTS_ON_GDK="$QA_RUN_TESTS_ON_GDK" --env QA_INFLUXDB_TOKEN="$QA_INFLUXDB_TOKEN" --env QA_TEST_LICENSE_MODE="$QA_TEST_LICENSE_MODE" --env QA_TEST_SESSION_TOKEN="$QA_TEST_SESSION_TOKEN" --env QA_SLOW_CONNECTION_LATENCY_MS="$QA_SLOW_CONNECTION_LATENCY_MS" --env QA_RESULTS_ISSUE_PROJECT_TOKEN="$QA_RESULTS_ISSUE_PROJECT_TOKEN" --env QA_GITHUB_ACCESS_TOKEN="$QA_GITHUB_ACCESS_TOKEN" --env QA_FACEBOOK_USERNAME="$QA_FACEBOOK_USERNAME" --env QA_OMNIBUS_MR_TESTS="$QA_OMNIBUS_MR_TESTS" --env QA_THIRD_PARTY_DOCKER_REPOSITORY="$QA_THIRD_PARTY_DOCKER_REPOSITORY" --env QA_SELENOID_BROWSER_VERSION="111.0" --env QA_VIDEO_RECORDER_IMAGE="presidenten/selenoid-manual-video-recorder" --env QA_VIDEO_RECORDER_VERSION="latest" --env QA_SELENOID_BROWSER_IMAGE="selenoid/chrome" --env GITLAB_QA_ADMIN_ACCESS_TOKEN="$GITLAB_QA_ADMIN_ACCESS_TOKEN" --env GITLAB_QA_PASSWORD_1="$GITLAB_QA_PASSWORD_1" --env GITLAB_QA_PASSWORD_2="$GITLAB_QA_PASSWORD_2" --env KNAPSACK_GENERATE_REPORT="$KNAPSACK_GENERATE_REPORT" --env RSPEC_FAST_QUARANTINE_PATH="$RSPEC_FAST_QUARANTINE_PATH" --env RSPEC_SKIPPED_TESTS_REPORT_PATH="$RSPEC_SKIPPED_TESTS_REPORT_PATH" --env CI="$CI" --env CI_JOB_NAME="$CI_JOB_NAME" --env CI_JOB_NAME_SLUG="$CI_JOB_NAME_SLUG" --env CI_JOB_URL="$CI_JOB_URL" --env CI_JOB_TOKEN="$CI_JOB_TOKEN" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env CI_PROJECT_NAME="$CI_PROJECT_NAME" --env CI_PROJECT_PATH="$CI_PROJECT_PATH" --env CI_PIPELINE_SOURCE="$CI_PIPELINE_SOURCE" --env CI_PIPELINE_URL="$CI_PIPELINE_URL" --env CI_PIPELINE_CREATED_AT="$CI_PIPELINE_CREATED_AT" --env GITLAB_CI="$GITLAB_CI" --env GITLAB_QA_USER_AGENT="$GITLAB_QA_USER_AGENT" --env GITLAB_INITIAL_ROOT_PASSWORD="5iveL!fe" --env COLORIZED_LOGS="$COLORIZED_LOGS" --env RELEASE="$RELEASE" --env RELEASE_REGISTRY_URL="registry.gitlab.com" --env RELEASE_REGISTRY_USERNAME="gitlab-ci-token" --env RELEASE_REGISTRY_PASSWORD="[MASKED]" --env SELENOID_DIRECTORY="/builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/selenoid" --env SCHEDULE_TYPE="$SCHEDULE_TYPE" --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-12-05-23-58-12-3f484a7f/gitlab-ee-qa-497583bb:/home/gitlab/qa/tmp:z --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /builds/gitlab-org/gitlab/qa/rspec:/home/gitlab/qa/rspec:z --name gitlab-ee-qa-497583bb registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:807c767b70d28221a50c9acb95390fece870a5c5 Test::Instance::All http://gitlab-gitaly-cluster.test -- --format QA::Support::JsonFormatter --out tmp/rspec-5688711924.json --format RspecJunitFormatter --out tmp/rspec-5688711924.xml --format html --out tmp/rspec-gitaly-cluster.htm --color --format documentation --tag gitaly_cluster`
628[Dec 06 2023 00:04:59 UTC (QA Tests)] WARN -- /home/gitlab/qa//home/gitlab/qa/rspec/fast_quarantine-gitlab.txt doesn't exist!
6292023-12-06 00:05:00 / CONF ::
630 ==> Base URL: http://gitlab-gitaly-cluster.test
631 ==> Browser: #<Selenium::WebDriver::Chrome::Driver:0x00007ffb689dd738>
632 ==> Libraries: Chemlab::Vendor
633Run options:
634 include {:gitaly_cluster=>true}
635 exclude {:geo=>true, :requires_praefect=>true}
636Randomized with seed 22214
637[Dec 06 2023 00:06:27 UTC (QA Tests)] INFO -- Caching token for username: root, last six chars of token:sixwPA
638[Dec 06 2023 00:06:27 UTC (QA Tests)] INFO -- Using Browser: chrome
639[Dec 06 2023 00:06:27 UTC (QA Tests)] INFO -- Performing sanity check for environment!
640[Dec 06 2023 00:06:28 UTC (QA Tests)] INFO -- Performing initial license fabrication!
641[Dec 06 2023 00:06:28 UTC (QA Tests)] INFO -- Successfully added license key. Details:
642{:test_license_mode=>false, :license_key_length=>2136, :last_five_characters=>"cbiJ9", :plan=>"ultimate", :starts_at=>"2023-02-03", :user_limit=>10000}
643[Dec 06 2023 00:06:28 UTC (QA Tests)] INFO -- ==> Built a QA::EE::Resource::License with id '1' via api in 0.35 seconds
644[Dec 06 2023 00:06:28 UTC (QA Tests)] INFO -- Disabling sync with External package metadata database
645[Dec 06 2023 00:06:29 UTC (QA Tests)] INFO -- Setting application settings: {:package_metadata_purl_types=>[11]}
646Systems
647 Gitaly Cluster replication queue
648No such file or directory @ rb_sysopen - /home/gitlab/qa//home/gitlab/qa/rspec/fast_quarantine-gitlab.txt
649[Dec 06 2023 00:06:29 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly Cluster replication queue allows replication of different repository after interruption
650[Dec 06 2023 00:06:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
651[Dec 06 2023 00:06:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
652`
653[Dec 06 2023 00:06:30 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
654[Dec 06 2023 00:06:30 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
655[Dec 06 2023 00:06:30 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
656[Dec 06 2023 00:06:30 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
657[Dec 06 2023 00:06:30 UTC (QA Tests)] INFO -- Waiting for health check on praefect
658[Dec 06 2023 00:06:30 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
659[Dec 06 2023 00:06:31 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
660[Dec 06 2023 00:06:31 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
661[Dec 06 2023 00:06:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
662`
663[Dec 06 2023 00:06:32 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
664[Dec 06 2023 00:06:32 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
665[Dec 06 2023 00:06:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
666`
667[Dec 06 2023 00:06:34 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
668[Dec 06 2023 00:06:34 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
669[Dec 06 2023 00:06:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
670`
671[Dec 06 2023 00:06:35 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
672[Dec 06 2023 00:06:35 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
673[Dec 06 2023 00:06:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
674`
675[Dec 06 2023 00:06:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
676`
677[Dec 06 2023 00:06:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
678`
679[Dec 06 2023 00:06:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
680`
681[Dec 06 2023 00:06:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
682`
683[Dec 06 2023 00:06:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
684`
685[Dec 06 2023 00:06:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
686`
687[Dec 06 2023 00:06:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
688`
689[Dec 06 2023 00:06:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
690`
691[Dec 06 2023 00:06:46 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
692`
693[Dec 06 2023 00:06:46 UTC (QA Tests)] INFO -- Retrieved cached token for username: root, last six chars of token:sixwPA
694[Dec 06 2023 00:06:46 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::PersonalAccessToken with name 'api-pat-root-4h61fydk' via cache in 0.0 seconds
695[Dec 06 2023 00:06:47 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-4' via api in 1.14 seconds
696[Dec 06 2023 00:06:49 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e' via api in 2.64 seconds
697[Dec 06 2023 00:06:52 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e/gitaly_cluster-6b250cee7abfbe19' via api in 6.41 seconds
698[Dec 06 2023 00:07:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
699[Dec 06 2023 00:07:18 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
700[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
701[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
702`
703[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
704`
705[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
706`
707[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
708`
709[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
710`
711[Dec 06 2023 00:07:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
712`
713[Dec 06 2023 00:07:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
714`
715[Dec 06 2023 00:07:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
716`
717[Dec 06 2023 00:07:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
718`
719[Dec 06 2023 00:07:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
720`
721[Dec 06 2023 00:07:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
722`
723[Dec 06 2023 00:07:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
724`
725[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
726`
727[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
728`
729[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
730`
731[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
732`
733[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
734`
735[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
736`
737[Dec 06 2023 00:07:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
738`
739[Dec 06 2023 00:07:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
740`
741[Dec 06 2023 00:07:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
742`
743[Dec 06 2023 00:07:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
744`
745[Dec 06 2023 00:07:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
746`
747[Dec 06 2023 00:07:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
748`
749[Dec 06 2023 00:07:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
750`
751[Dec 06 2023 00:07:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
752`
753[Dec 06 2023 00:07:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
754`
755[Dec 06 2023 00:07:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
756`
757[Dec 06 2023 00:07:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
758`
759[Dec 06 2023 00:07:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
760`
761[Dec 06 2023 00:07:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
762`
763[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
764[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker pause praefect`
765[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Setting jobs in replication queue to `in_progress` and acquiring locks
766[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"update replication_queue set state = 'in_progress';
767insert into replication_queue_job_lock (job_id, lock_id, triggered_at)
768 select id, rq.lock_id, created_at from replication_queue rq
769 left join replication_queue_job_lock rqjl on rq.id = rqjl.job_id
770 where state = 'in_progress' and rqjl.job_id is null;
771update replication_queue_lock set acquired = 't';
772\""
773`
774[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
775[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker unpause praefect`
776[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} praefect`
777[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Waiting for health check on praefect
778[Dec 06 2023 00:07:24 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
779[Dec 06 2023 00:07:25 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-4' via api in 0.07 seconds
780[Dec 06 2023 00:07:26 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e' via api in 0.21 seconds
781[Dec 06 2023 00:07:28 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e/the_awesome_project-4a1037ea0603051b' via api in 2.71 seconds
782[Dec 06 2023 00:07:28 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[2]"'`
783[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
784[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
785`
786[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
787[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
788[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
789[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
790[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Waiting for health check on praefect
791[Dec 06 2023 00:07:58 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
792[Dec 06 2023 00:07:59 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
793[Dec 06 2023 00:07:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
794[Dec 06 2023 00:08:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
795`
796[Dec 06 2023 00:08:01 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
797[Dec 06 2023 00:08:01 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
798[Dec 06 2023 00:08:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
799`
800[Dec 06 2023 00:08:02 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
801[Dec 06 2023 00:08:02 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
802[Dec 06 2023 00:08:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
803`
804[Dec 06 2023 00:08:03 UTC (QA Tests)] INFO -- Clearing the replication queue
805[Dec 06 2023 00:08:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from replication_queue_job_lock;
806delete from replication_queue_lock;
807delete from replication_queue;
808\""
809`
810 allows replication of different repository after interruption
811Systems
812 Gitaly distributed reads
813[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly distributed reads reads from each node
814[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
815[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
816`
817[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
818[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
819[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
820[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
821[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Waiting for health check on praefect
822[Dec 06 2023 00:08:04 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
823[Dec 06 2023 00:08:05 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
824[Dec 06 2023 00:08:05 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
825[Dec 06 2023 00:08:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
826`
827[Dec 06 2023 00:08:07 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
828[Dec 06 2023 00:08:07 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
829[Dec 06 2023 00:08:08 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
830`
831[Dec 06 2023 00:08:08 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
832[Dec 06 2023 00:08:08 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
833[Dec 06 2023 00:08:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
834`
835[Dec 06 2023 00:08:09 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-4' via api in 0.08 seconds
836[Dec 06 2023 00:08:10 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e' via api in 0.2 seconds
837[Dec 06 2023 00:08:12 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e/gitaly_cluster-971dabc020f5cdf8' via api in 2.48 seconds
838[Dec 06 2023 00:08:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
839`
840[Dec 06 2023 00:08:12 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`
841[Dec 06 2023 00:08:42 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
842[Dec 06 2023 00:08:42 UTC (QA Tests)] INFO -- Reading from the repository
843[Dec 06 2023 00:08:43 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
844[Dec 06 2023 00:08:48 UTC (QA Tests)] INFO -- Reading from the repository
845[Dec 06 2023 00:08:50 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
846[Dec 06 2023 00:08:55 UTC (QA Tests)] INFO -- Reading from the repository
847[Dec 06 2023 00:08:56 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
848[Dec 06 2023 00:08:56 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
849 reads from each node
850 when a node is unhealthy
851[Dec 06 2023 00:08:56 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly distributed reads when a node is unhealthy does not read from the unhealthy node
852[Dec 06 2023 00:08:56 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
853[Dec 06 2023 00:08:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
854`
855[Dec 06 2023 00:08:57 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
856[Dec 06 2023 00:08:57 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
857[Dec 06 2023 00:08:57 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
858[Dec 06 2023 00:08:57 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
859[Dec 06 2023 00:08:57 UTC (QA Tests)] INFO -- Waiting for health check on praefect
860[Dec 06 2023 00:08:57 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
861[Dec 06 2023 00:08:58 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
862[Dec 06 2023 00:08:58 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
863[Dec 06 2023 00:08:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
864`
865[Dec 06 2023 00:08:59 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
866[Dec 06 2023 00:08:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
867[Dec 06 2023 00:09:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
868`
869[Dec 06 2023 00:09:01 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
870[Dec 06 2023 00:09:01 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
871[Dec 06 2023 00:09:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
872`
873[Dec 06 2023 00:09:02 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-4' via api in 0.1 seconds
874[Dec 06 2023 00:09:02 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e' via api in 0.23 seconds
875[Dec 06 2023 00:09:04 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e/gitaly_cluster-871a9485777e26db' via api in 2.38 seconds
876[Dec 06 2023 00:09:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
877`
878[Dec 06 2023 00:09:04 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
879[Dec 06 2023 00:09:34 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
880[Dec 06 2023 00:09:34 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
881[Dec 06 2023 00:09:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
882`
883[Dec 06 2023 00:09:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
884`
885[Dec 06 2023 00:09:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
886`
887[Dec 06 2023 00:09:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
888`
889[Dec 06 2023 00:09:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
890`
891[Dec 06 2023 00:09:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
892`
893[Dec 06 2023 00:09:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
894`
895[Dec 06 2023 00:09:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
896`
897[Dec 06 2023 00:09:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
898`
899[Dec 06 2023 00:09:44 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
900[Dec 06 2023 00:09:44 UTC (QA Tests)] INFO -- Reading from the repository
901[Dec 06 2023 00:09:50 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
902[Dec 06 2023 00:09:51 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
903[Dec 06 2023 00:09:52 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
904[Dec 06 2023 00:09:53 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
905[Dec 06 2023 00:09:53 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
906[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
907[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
908[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
909 does not read from the unhealthy node
910Systems
911 Gitaly backend node recovery
912[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly backend node recovery recovers from dataloss
913[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
914[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
915`
916[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
917[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
918[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
919[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
920[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Waiting for health check on praefect
921[Dec 06 2023 00:09:54 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
922[Dec 06 2023 00:09:56 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
923[Dec 06 2023 00:09:56 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
924[Dec 06 2023 00:09:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
925`
926[Dec 06 2023 00:09:57 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
927[Dec 06 2023 00:09:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
928[Dec 06 2023 00:09:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
929`
930[Dec 06 2023 00:09:58 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
931[Dec 06 2023 00:09:58 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
932[Dec 06 2023 00:10:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
933`
934[Dec 06 2023 00:10:00 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-4' via api in 0.15 seconds
935[Dec 06 2023 00:10:00 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e' via api in 0.41 seconds
936[Dec 06 2023 00:10:03 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e/gitaly_cluster-516e1398d85773f8' via api in 3.37 seconds
937[Dec 06 2023 00:10:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
938`
939[Dec 06 2023 00:10:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[5]"'`
940[Dec 06 2023 00:10:34 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
941[Dec 06 2023 00:10:34 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`
942[Dec 06 2023 00:10:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
943`
944[Dec 06 2023 00:10:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
945`
946[Dec 06 2023 00:10:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
947`
948[Dec 06 2023 00:10:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
949`
950[Dec 06 2023 00:10:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
951`
952[Dec 06 2023 00:10:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
953`
954[Dec 06 2023 00:10:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
955`
956[Dec 06 2023 00:10:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
957`
958[Dec 06 2023 00:10:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
959`
960[Dec 06 2023 00:10:43 UTC (QA Tests)] INFO -- Waiting for health check failure on gitaly1
961[Dec 06 2023 00:10:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
962`
963[Dec 06 2023 00:10:46 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "pushed after failover"
964[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 3.28 seconds
965[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select job from replication_queue
966where state = 'ready'
967 and job ->> 'change' = 'update'
968 and job ->> 'target_node_storage' = 'gitaly1';
969\""
970`
971[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
972[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`
973[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
974[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
975[Dec 06 2023 00:10:47 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
976[Dec 06 2023 00:10:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
977`
978[Dec 06 2023 00:10:49 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
979[Dec 06 2023 00:10:49 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
980[Dec 06 2023 00:10:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
981`
982[Dec 06 2023 00:10:50 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
983[Dec 06 2023 00:10:50 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
984[Dec 06 2023 00:10:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
985`
986[Dec 06 2023 00:10:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
987`
988[Dec 06 2023 00:10:52 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[5]"'`
989[Dec 06 2023 00:11:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
990[Dec 06 2023 00:11:23 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
991[Dec 06 2023 00:11:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
992`
993[Dec 06 2023 00:11:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
994`
995[Dec 06 2023 00:11:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
996`
997[Dec 06 2023 00:11:26 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
998`
999[Dec 06 2023 00:11:28 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1000`
1001[Dec 06 2023 00:11:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1002`
1003[Dec 06 2023 00:11:30 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1004`
1005[Dec 06 2023 00:11:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1006`
1007[Dec 06 2023 00:11:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1008`
1009[Dec 06 2023 00:11:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1010`
1011[Dec 06 2023 00:11:34 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
1012[Dec 06 2023 00:11:34 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`
1013[Dec 06 2023 00:11:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1014`
1015[Dec 06 2023 00:11:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1016`
1017[Dec 06 2023 00:11:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1018`
1019[Dec 06 2023 00:11:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1020`
1021[Dec 06 2023 00:11:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1022`
1023[Dec 06 2023 00:11:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1024`
1025[Dec 06 2023 00:11:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1026`
1027[Dec 06 2023 00:11:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1028`
1029[Dec 06 2023 00:11:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1030`
1031[Dec 06 2023 00:11:43 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
1032[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
1033`
1034[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
1035[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
1036[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
1037[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
1038[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
1039[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`
1040[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
1041[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
1042[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Waiting for health check on praefect
1043[Dec 06 2023 00:11:44 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
1044[Dec 06 2023 00:11:46 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
1045[Dec 06 2023 00:11:46 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
1046[Dec 06 2023 00:11:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1047`
1048[Dec 06 2023 00:11:47 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
1049[Dec 06 2023 00:11:47 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
1050[Dec 06 2023 00:11:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1051`
1052[Dec 06 2023 00:11:48 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
1053[Dec 06 2023 00:11:48 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
1054[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1055`
1056 recovers from dataloss
1057Systems
1058 with Gitaly automatic failover and recovery
1059[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
1060[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
1061`
1062[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
1063[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
1064[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
1065[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
1066[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Waiting for health check on praefect
1067[Dec 06 2023 00:11:50 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
1068[Dec 06 2023 00:11:52 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
1069[Dec 06 2023 00:11:52 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
1070[Dec 06 2023 00:11:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1071`
1072[Dec 06 2023 00:11:53 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
1073[Dec 06 2023 00:11:53 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
1074[Dec 06 2023 00:11:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1075`
1076[Dec 06 2023 00:11:54 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
1077[Dec 06 2023 00:11:54 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
1078[Dec 06 2023 00:11:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1079`
1080[Dec 06 2023 00:11:56 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-4' via api in 0.08 seconds
1081[Dec 06 2023 00:11:56 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e' via api in 0.21 seconds
1082[Dec 06 2023 00:11:59 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/qa-test-2023-12-06-00-06-46-66384cfd1ac4bf4e/gitaly_cluster-2208e530d6c3b921' via api in 3.08 seconds
1083[Dec 06 2023 00:11:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
1084`
1085[Dec 06 2023 00:11:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[6]"'`
1086[Dec 06 2023 00:12:29 UTC (QA Tests)] INFO -- Starting test: Systems with Gitaly automatic failover and recovery automatically fails over
1087[Dec 06 2023 00:12:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
1088[Dec 06 2023 00:12:29 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
1089[Dec 06 2023 00:12:30 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1090`
1091[Dec 06 2023 00:12:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1092`
1093[Dec 06 2023 00:12:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1094`
1095[Dec 06 2023 00:12:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1096`
1097[Dec 06 2023 00:12:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1098`
1099[Dec 06 2023 00:12:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1100`
1101[Dec 06 2023 00:12:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1102`
1103[Dec 06 2023 00:12:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1104`
1105[Dec 06 2023 00:12:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1106`
1107[Dec 06 2023 00:12:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1108`
1109[Dec 06 2023 00:12:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
1110[Dec 06 2023 00:12:40 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`
1111[Dec 06 2023 00:12:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1112`
1113[Dec 06 2023 00:12:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1114`
1115[Dec 06 2023 00:12:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1116`
1117[Dec 06 2023 00:12:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1118`
1119[Dec 06 2023 00:12:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1120`
1121[Dec 06 2023 00:12:46 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1122`
1123[Dec 06 2023 00:12:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1124`
1125[Dec 06 2023 00:12:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1126`
1127[Dec 06 2023 00:12:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1128`
1129[Dec 06 2023 00:12:52 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "first_added_commit_message to primary gitaly node"
1130[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.81 seconds
1131[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
1132[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
1133`
1134[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
1135[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
1136[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
1137[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
1138[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
1139[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`
1140[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
1141[Dec 06 2023 00:12:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
1142[Dec 06 2023 00:12:54 UTC (QA Tests)] INFO -- Waiting for health check on praefect
1143[Dec 06 2023 00:12:54 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
1144[Dec 06 2023 00:12:55 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
1145[Dec 06 2023 00:12:55 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
1146[Dec 06 2023 00:12:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1147`
1148[Dec 06 2023 00:12:56 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
1149[Dec 06 2023 00:12:56 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
1150[Dec 06 2023 00:12:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1151`
1152[Dec 06 2023 00:12:58 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
1153[Dec 06 2023 00:12:58 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
1154[Dec 06 2023 00:12:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1155`
1156[Dec 06 2023 00:12:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
1157`
1158[Dec 06 2023 00:12:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[6]"'`
1159[Dec 06 2023 00:13:30 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
1160[Dec 06 2023 00:13:30 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`
1161[Dec 06 2023 00:13:30 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1162`
1163[Dec 06 2023 00:13:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1164`
1165[Dec 06 2023 00:13:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1166`
1167[Dec 06 2023 00:13:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1168`
1169[Dec 06 2023 00:13:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1170`
1171[Dec 06 2023 00:13:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1172`
1173[Dec 06 2023 00:13:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1174`
1175[Dec 06 2023 00:13:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1176`
1177[Dec 06 2023 00:13:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1178`
1179[Dec 06 2023 00:13:40 UTC (QA Tests)] INFO -- Waiting for health check failure on gitaly1
1180[Dec 06 2023 00:13:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1181`
1182 automatically fails over
1183 when recovering from dataloss after failover
1184[Dec 06 2023 00:13:42 UTC (QA Tests)] INFO -- Starting test: Systems with Gitaly automatic failover and recovery when recovering from dataloss after failover automatically reconciles
1185[Dec 06 2023 00:13:42 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
1186[Dec 06 2023 00:13:42 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`
1187[Dec 06 2023 00:13:42 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
1188[Dec 06 2023 00:13:42 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
1189[Dec 06 2023 00:13:42 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
1190[Dec 06 2023 00:13:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1191`
1192[Dec 06 2023 00:13:43 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[6]"'`
1193[Dec 06 2023 00:14:14 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
1194[Dec 06 2023 00:14:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
1195`
1196[Dec 06 2023 00:14:14 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
1197[Dec 06 2023 00:14:15 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
1198[Dec 06 2023 00:14:15 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
1199[Dec 06 2023 00:14:15 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
1200[Dec 06 2023 00:14:15 UTC (QA Tests)] INFO -- Waiting for health check on praefect
1201[Dec 06 2023 00:14:15 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
1202[Dec 06 2023 00:14:16 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
1203[Dec 06 2023 00:14:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
1204[Dec 06 2023 00:14:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
1205`
1206[Dec 06 2023 00:14:17 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
1207[Dec 06 2023 00:14:17 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
1208[Dec 06 2023 00:14:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
1209`
1210[Dec 06 2023 00:14:19 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
1211[Dec 06 2023 00:14:19 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
1212[Dec 06 2023 00:14:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
1213`
1214 automatically reconciles
1215Finished in 9 minutes 19 seconds (files took 2.41 seconds to load)
12166 examples, 0 failures
1217Randomized with seed 22214
1218[Dec 06 2023 00:14:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select n_live_tup, n_dead_tup, relname from pg_stat_all_tables order by n_live_tup DESC, n_dead_tup DESC;'"`
1219[Dec 06 2023 00:14:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select * from pg_stat_user_tables;'"`
1220[Dec 06 2023 00:14:21 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=gitlab-gitaly-cluster`
1221[Dec 06 2023 00:14:21 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f gitlab-gitaly-cluster`
1222 Running after_script 00:21
1223Running after script...
1224$ if [ "$CI_JOB_STATUS" == "failed" ]; then # collapsed multi-line command
1225$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
1226$ echo -e "\e[0Ksection_start:`date +%s`:report_results_section[collapsed=true]\r\e[0KReport results"
1227 Report results 00:20
1249 Saving cache for successful job 00:00
1250Not uploading cache qa-e2e-ruby-3.0-170e3eb0aa6d1cf29b552c828433d8e0-16 due to policy
1251 Uploading artifacts for successful job 00:03
1252Uploading artifacts...
1253gitlab-qa-run-*: found 220 matching artifact files and directories
1254Uploading artifacts as "archive" to coordinator... 201 Created id=5688711924 responseStatus=201 Created token=64_GspgH
1255Uploading artifacts...
1256gitlab-qa-run-*/**/rspec-*.xml: found 1 matching artifact files and directories
1257Uploading artifacts as "junit" to coordinator... 201 Created id=5688711924 responseStatus=201 Created token=64_GspgH
1258Uploading artifacts...
1259/builds/gitlab-org/gitlab/suite_status.env: found 1 matching artifact files and directories
1260Uploading artifacts as "dotenv" to coordinator... 201 Created id=5688711924 responseStatus=201 Created token=64_GspgH
1261 Cleaning up project directory and file based variables 00:00
1262Job succeeded