gitaly-cluster
Passed Started
by
@DylanGriffith

Dylan Griffith
1Running with gitlab-runner 15.3.0 (bbcb5aba)2 on gitlab-org-qa-runner ss7o3c7p5Using Docker executor with image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 ...6Starting service docker:23.0.1-dind ...7Pulling docker image docker:23.0.1-dind ...8Using docker image sha256:0965b14b96d892eee2a963801769af5328817a64ed1af67ec4d76364ddd62706 for docker:23.0.1-dind with digest docker@sha256:d9a0fd8bdd156aae6e9be5db0264e0bc99e000e89e92055d4567e27ff13daff0 ...9Waiting for services to be up and running (timeout 30 seconds)...10Authenticating with credentials from job payload (GitLab Registry)11Pulling docker image registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 ...12Using docker image sha256:cd8bbceb0e95190ac7f7e9b1e6e2e0f6e16a7aa683006352e1152901402c5523 for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:1417fb91ae6091da52a3d17144fa553ea56ee6f3b46e2029e9e959da02b99bde ...14Running on runner-ss7o3c7p-project-278964-concurrent-0 via runner-ss7o3c7p-gitlab-org-qa-1683233875-b9f71a9d-gcp...16Fetching changes with git depth set to 20...17Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/18Created fresh repository.19remote: Enumerating objects: 86055, done. 20remote: Counting objects: 100% (86055/86055), done. 21remote: Compressing objects: 100% (68977/68977), done. 22remote: Total 86055 (delta 19305), reused 52259 (delta 13123), pack-reused 0 23Receiving objects: 100% (86055/86055), 104.27 MiB | 25.12 MiB/s, done.24Resolving deltas: 100% (19305/19305), done.26 * [new ref] refs/pipelines/857810750 -> refs/pipelines/85781075027remote: Enumerating objects: 5, done. 28remote: Counting objects: 100% (5/5), done. 29remote: Compressing objects: 100% (5/5), done. 30remote: Total 5 (delta 0), reused 5 (delta 0), pack-reused 0 31 * [new tag] v15.11.0-ee -> v15.11.0-ee32 * [new tag] v15.11.0-rc42-ee -> v15.11.0-rc42-ee33 * [new tag] v15.11.0-rc43-ee -> v15.11.0-rc43-ee34 * [new tag] v15.11.0-rc44-ee -> v15.11.0-rc44-ee35 * [new tag] v15.11.1-ee -> v15.11.1-ee36Checking out 53f90aa5 as refs/merge-requests/119520/merge...37Skipping Git submodules setup39Checking cache for qa-e2e-ruby-3.0-c9589dbb96af39a99372100e42ffffe8-16...40Downloading cache.zip from https://storage.googleapis.com/gitlab-qa-runner-cache-1/project/278964/qa-e2e-ruby-3.0-c9589dbb96af39a99372100e42ffffe8-16 41WARNING: .gems/gems/deprecation_toolkit-2.0.3/gemfiles/test/deprecations: chmod .gems/gems/deprecation_toolkit-2.0.3/gemfiles/test/deprecations: no such file or directory (suppressing repeats) 42Successfully extracted cache44Downloading artifacts for download-knapsack-report (4230518589)...45Downloading artifacts from coordinator... ok id=4230518589 responseStatus=200 OK token=64_Ucp7b47Using docker image sha256:cd8bbceb0e95190ac7f7e9b1e6e2e0f6e16a7aa683006352e1152901402c5523 for registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0:bundler-2.3-docker-23.0.1 with digest registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-3.0@sha256:1417fb91ae6091da52a3d17144fa553ea56ee6f3b46e2029e9e959da02b99bde ...48$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")49$ if [ "$RUN_WITH_BUNDLE" == "true" ]; then # collapsed multi-line command50Bundle complete! 36 Gemfile dependencies, 128 gems now installed.51Use `bundle info [gemname]` to see where a bundled gem is installed.52$ export QA_COMMAND="$BUNDLE_PREFIX gitlab-qa ${QA_SCENARIO:=Test::Instance::Image} $RELEASE $GITLAB_QA_OPTS -- $QA_TESTS $QA_RSPEC_TAGS $RSPEC_REPORT_OPTS"53$ echo "Running - '$QA_COMMAND'"54Running - 'bundle exec gitlab-qa Test::Integration::GitalyCluster registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638 -- --format QA::Support::JsonFormatter --out tmp/rspec-4230518628.json --format RspecJunitFormatter --out tmp/rspec-4230518628.xml --format html --out tmp/rspec-$(echo gitaly-cluster | sed 's/[ /]/_/g').htm --color --format documentation'55$ eval "$QA_COMMAND"56[May 05 2023 00:43:11 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q telegraf:1.21-alpine`57[May 05 2023 00:43:13 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^telegraf$`58[May 05 2023 00:43:13 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name telegraf --user root --entrypoint telegraf --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/telegraf:/etc/telegraf/log:z --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /builds/d20230505-21-qeimzh/telegraf.conf:/etc/telegraf/telegraf.conf:ro --env QA_INFLUXDB_TOKEN="$QA_INFLUXDB_TOKEN" --env QA_INFLUXDB_URL="$QA_INFLUXDB_URL" --env QA_RUN_TYPE="$QA_RUN_TYPE" --env CI_JOB_NAME="$CI_JOB_NAME" --env CI_PIPELINE_ID="$CI_PIPELINE_ID" telegraf:1.21-alpine`59[May 05 2023 00:43:15 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q postgres:11`60[May 05 2023 00:43:19 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^postgres$`61[May 05 2023 00:43:19 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`62[May 05 2023 00:43:19 UTC (Gitlab QA)] INFO -- Shell command: `docker network create test`63[May 05 2023 00:43:19 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name postgres --net test --env POSTGRES_PASSWORD="SQL_PASSWORD" postgres:11`64[May 05 2023 00:43:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`65[May 05 2023 00:43:21 UTC (Gitlab QA)] ERROR -- Shell command output:66psql: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory67 Is the server running locally and accepting connections on that socket?68[May 05 2023 00:43:26 UTC (Gitlab QA)] INFO -- Shell command: `docker exec postgres bash -c "psql -U postgres template1"`69[May 05 2023 00:43:27 UTC (Gitlab QA)] INFO -- Shell command: `docker exec postgres bash -c "psql -U postgres -d template1 -c \"CREATE DATABASE praefect_production OWNER postgres\""`70[May 05 2023 00:43:27 UTC (Gitlab QA)] INFO -- Starting Gitaly Cluster71[May 05 2023 00:43:27 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^alpine-c82a5236$`72[May 05 2023 00:43:27 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`73[May 05 2023 00:43:27 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name alpine-c82a5236 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z alpine:latest tail -f /dev/null`74[May 05 2023 00:43:27 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`75[May 05 2023 00:43:33 UTC (Gitlab QA)] INFO -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-10.3.0/tls_certificates/authority/. alpine-c82a5236:/etc/gitlab/trusted-certs`76[May 05 2023 00:43:34 UTC (Gitlab QA)] INFO -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-10.3.0/tls_certificates/gitlab/. alpine-c82a5236:/etc/gitlab/ssl`77[May 05 2023 00:43:34 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=alpine-c82a5236`78[May 05 2023 00:43:34 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f alpine-c82a5236`79[May 05 2023 00:43:37 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`80[May 05 2023 00:43:37 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`81[May 05 2023 00:44:02 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitlab-gitaly-cluster$`82[May 05 2023 00:44:02 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly1$`83[May 05 2023 00:44:02 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`84[May 05 2023 00:44:02 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`85[May 05 2023 00:44:02 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly1 --net test --hostname gitaly1.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/gitaly1/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`86[May 05 2023 00:44:02 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitlab-gitaly-cluster --net test --hostname gitlab-gitaly-cluster.test -p 80 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/gitlab-gitaly-cluster/logs:/var/log/gitlab:Z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`87[May 05 2023 00:44:20 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "echo \"postgresql['enable'] = false;88redis['enable'] = false;89nginx['enable'] = false;90grafana['enable'] = false;91puma['enable'] = false;92sidekiq['enable'] = false;93gitlab_workhorse['enable'] = false;94gitlab_rails['rake_cache_clear'] = false;95gitlab_rails['auto_migrate'] = false;96gitlab_exporter['enable'] = false;97praefect['enable'] = false;98prometheus['enable'] = true;99gitaly['enable'] = true;100gitaly['listen_addr'] = '0.0.0.0:8075';101gitaly['prometheus_listen_addr'] = '0.0.0.0:9236';102gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN';103gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';104gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';105git_data_dirs({106 'gitaly1' => {107 'path' => '/var/opt/gitlab/git-data'108 },109 'gitaly2' => {110 'path' => '/var/opt/gitlab/git-data'111 },112 'gitaly3' => {113 'path' => '/var/opt/gitlab/git-data'114 }115});116\" > /etc/gitlab/gitlab.rb;"`117[May 05 2023 00:44:20 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo \"# Generated by GitLab QA Omnibus Configurator at 2023-05-05 00:43:15118# Gitlab::QA::Runtime::OmnibusConfigurations::Default119gitlab_rails['gitlab_default_theme'] = 10 # Light Red Theme120gitlab_rails['gitlab_disable_animations'] = true # Disable animations121gitlab_rails['application_settings_cache_seconds'] = 0 # Settings cache expiry122gitlab_rails['initial_root_password'] = '5iveL!fe' # Initial root password123external_url 'http://gitlab-gitaly-cluster.test';124git_data_dirs({125 'default' => {126 'gitaly_address' => 'tcp://praefect.test:2305',127 'gitaly_token' => 'PRAEFECT_EXTERNAL_TOKEN'128 }129});130gitaly['enable'] = false;131gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';132prometheus['scrape_configs'] = [133 {134 'job_name' => 'praefect',135 'static_configs' => [136 'targets' => [137 'praefect.test:9652'138 ]139 ]140 },141 {142 'job_name' => 'praefect-gitaly',143 'static_configs' => [144 'targets' => [145 'gitaly1.test:9236',146 'gitaly2.test:9236',147 'gitaly3.test:9236'148 ]149 ]150 }151];152grafana['disable_login_form'] = false;153grafana['admin_password'] = 'GRAFANA_ADMIN_PASSWORD';154gitlab_rails['env'] = {'GITLAB_LICENSE_MODE'=>'test', 'CUSTOMER_PORTAL_URL'=>'https://customers.staging.gitlab.com', 'GITLAB_QA_USER_AGENT'=>'*****', 'GITLAB_ALLOW_SEPARATE_CI_DATABASE'=>'true'}\" > /etc/gitlab/gitlab.rb;"`155[May 05 2023 00:44:20 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly1`156[May 05 2023 00:44:20 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitlab-gitaly-cluster`157[May 05 2023 00:45:19 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`158[May 05 2023 00:45:19 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "echo '#!/usr/bin/env bash159if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then160 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'161 exit 1162fi163' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`164[May 05 2023 00:45:19 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly1 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`165[May 05 2023 00:45:19 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`166[May 05 2023 00:45:19 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`167[May 05 2023 00:45:19 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly2$`168[May 05 2023 00:45:20 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`169[May 05 2023 00:45:20 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly2 --net test --hostname gitaly2.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/gitaly2/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`170[May 05 2023 00:45:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "echo \"postgresql['enable'] = false;171redis['enable'] = false;172nginx['enable'] = false;173grafana['enable'] = false;174puma['enable'] = false;175sidekiq['enable'] = false;176gitlab_workhorse['enable'] = false;177gitlab_rails['rake_cache_clear'] = false;178gitlab_rails['auto_migrate'] = false;179gitlab_exporter['enable'] = false;180praefect['enable'] = false;181prometheus['enable'] = true;182gitaly['enable'] = true;183gitaly['listen_addr'] = '0.0.0.0:8075';184gitaly['prometheus_listen_addr'] = '0.0.0.0:9236';185gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN';186gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';187gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';188git_data_dirs({189 'gitaly1' => {190 'path' => '/var/opt/gitlab/git-data'191 },192 'gitaly2' => {193 'path' => '/var/opt/gitlab/git-data'194 },195 'gitaly3' => {196 'path' => '/var/opt/gitlab/git-data'197 }198});199\" > /etc/gitlab/gitlab.rb;"`200[May 05 2023 00:45:21 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly2`201[May 05 2023 00:46:19 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`202[May 05 2023 00:46:19 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "echo '#!/usr/bin/env bash203if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then204 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'205 exit 1206fi207' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`208[May 05 2023 00:46:19 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly2 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`209[May 05 2023 00:46:19 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`210[May 05 2023 00:46:20 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`211[May 05 2023 00:46:20 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly3$`212[May 05 2023 00:46:20 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`213[May 05 2023 00:46:20 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly3 --net test --hostname gitaly3.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/gitaly3/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`214[May 05 2023 00:46:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "echo \"postgresql['enable'] = false;215redis['enable'] = false;216nginx['enable'] = false;217grafana['enable'] = false;218puma['enable'] = false;219sidekiq['enable'] = false;220gitlab_workhorse['enable'] = false;221gitlab_rails['rake_cache_clear'] = false;222gitlab_rails['auto_migrate'] = false;223gitlab_exporter['enable'] = false;224praefect['enable'] = false;225prometheus['enable'] = true;226gitaly['enable'] = true;227gitaly['listen_addr'] = '0.0.0.0:8075';228gitaly['prometheus_listen_addr'] = '0.0.0.0:9236';229gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN';230gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';231gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';232git_data_dirs({233 'gitaly1' => {234 'path' => '/var/opt/gitlab/git-data'235 },236 'gitaly2' => {237 'path' => '/var/opt/gitlab/git-data'238 },239 'gitaly3' => {240 'path' => '/var/opt/gitlab/git-data'241 }242});243\" > /etc/gitlab/gitlab.rb;"`244[May 05 2023 00:46:21 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly3`245[May 05 2023 00:47:20 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`246[May 05 2023 00:47:20 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "echo '#!/usr/bin/env bash247if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then248 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'249 exit 1250fi251' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`252[May 05 2023 00:47:20 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`253[May 05 2023 00:47:20 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`254[May 05 2023 00:47:21 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`255[May 05 2023 00:47:21 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^praefect$`256[May 05 2023 00:47:21 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`257[May 05 2023 00:47:21 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name praefect --net test --hostname praefect.test -p 2305 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/praefect/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:53f90aa57f8af288454aecc48b4a67b4ba12f638`258[May 05 2023 00:47:22 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "echo \"postgresql['enable'] = false;259redis['enable'] = false;260nginx['enable'] = false;261grafana['enable'] = false;262puma['enable'] = false;263sidekiq['enable'] = false;264gitlab_workhorse['enable'] = false;265gitlab_rails['rake_cache_clear'] = false;266gitlab_rails['auto_migrate'] = false;267gitlab_exporter['enable'] = false;268gitaly['enable'] = false;269prometheus['enable'] = true;270praefect['enable'] = true;271praefect['listen_addr'] = '0.0.0.0:2305';272praefect['prometheus_listen_addr'] = '0.0.0.0:9652';273praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN';274praefect['reconciliation_scheduling_interval'] = '10s';275praefect['database_host'] = 'postgres.test';276praefect['database_user'] = 'postgres';277praefect['database_port'] = 5432;278praefect['database_password'] = 'SQL_PASSWORD';279praefect['database_dbname'] = 'praefect_production';280praefect['database_sslmode'] = 'disable';281praefect['database_direct_host'] = 'postgres.test';282praefect['database_direct_port'] = 5432;283praefect['virtual_storages'] = {284 'default' => {285 'nodes' => {286 'gitaly1' => {287 'address' => 'tcp://gitaly1.test:8075',288 'token' => 'PRAEFECT_INTERNAL_TOKEN'289 },290 'gitaly2' => {291 'address' => 'tcp://gitaly2.test:8075',292 'token' => 'PRAEFECT_INTERNAL_TOKEN'293 },294 'gitaly3' => {295 'address' => 'tcp://gitaly3.test:8075',296 'token' => 'PRAEFECT_INTERNAL_TOKEN'297 }298 }299 }300};301\" > /etc/gitlab/gitlab.rb;"`302[May 05 2023 00:47:22 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false praefect`303[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Waiting for GitLab to become healthy ...304[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-gitaly-cluster`305[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker port gitlab-gitaly-cluster 80/tcp`306[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- -> GitLab is available at `http://docker:32769`!307[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`308[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "echo '#!/usr/bin/env bash309if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then310 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'311 exit 1312fi313' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`314[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`315[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker cp /builds/gitlab-org/gitlab/.gems/gems/gitlab-qa-10.3.0/support/data gitlab-gitaly-cluster:/tmp/data-seeds`316[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Running exec_commands...317[May 05 2023 00:47:52 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-rails runner /tmp/data-seeds/admin_access_token_seed.rb"`318[May 05 2023 00:48:21 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "praefect -config /var/opt/gitlab/praefect/cluster_config.toml check || true"`319[May 05 2023 00:48:21 UTC (Gitlab QA)] INFO -- Shell command: `docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' praefect`320[May 05 2023 00:48:21 UTC (Gitlab QA)] INFO -- Gitaly Cluster Ready321[May 05 2023 00:48:42 UTC (Gitlab QA)] INFO -- Running Gitaly Cluster specs!322[May 05 2023 00:48:42 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`323[May 05 2023 00:48:43 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:53f90aa57f8af288454aecc48b4a67b4ba12f638`324[May 05 2023 00:49:42 UTC (Gitlab QA)] INFO -- Running test suite `Test::Instance::All` for gitlab-ee325[May 05 2023 00:49:42 UTC (Gitlab QA)] INFO -- Shell command: `docker run -t --rm --net=test --env QA_AWS_S3_REGION="$QA_AWS_S3_REGION" --env QA_GOOGLE_CDN_JSON_KEY="$QA_GOOGLE_CDN_JSON_KEY" --env QA_REPORT_RESULT_TOKEN="$QA_REPORT_RESULT_TOKEN" --env QA_THIRD_PARTY_DOCKER_REGISTRY="$QA_THIRD_PARTY_DOCKER_REGISTRY" --env QA_SAVE_TEST_METRICS="$QA_SAVE_TEST_METRICS" --env QA_GOOGLE_CDN_SIGNURL_KEY_NAME="$QA_GOOGLE_CDN_SIGNURL_KEY_NAME" --env QA_1P_PASSWORD="$QA_1P_PASSWORD" --env QA_GOOGLE_CDN_SIGNURL_KEY="$QA_GOOGLE_CDN_SIGNURL_KEY" --env QA_GITHUB_USERNAME="$QA_GITHUB_USERNAME" --env QA_GITHUB_PASSWORD="$QA_GITHUB_PASSWORD" --env QA_ARTIFACTS_DIR="$QA_ARTIFACTS_DIR" --env QA_GITLAB_TLS_CERTIFICATE="$QA_GITLAB_TLS_CERTIFICATE" --env QA_RUN_TYPE="$QA_RUN_TYPE" --env QA_JIRA_ADMIN_PASSWORD="$QA_JIRA_ADMIN_PASSWORD" --env QA_GITHUB_OAUTH_APP_SECRET="$QA_GITHUB_OAUTH_APP_SECRET" --env QA_PATH="$QA_PATH" --env QA_JIRA_ADMIN_USERNAME="$QA_JIRA_ADMIN_USERNAME" --env QA_ADMIN_ACCESS_TOKEN="$QA_ADMIN_ACCESS_TOKEN" --env QA_COMMAND="$QA_COMMAND" --env QA_EE_LICENSE="$QA_EE_LICENSE" --env QA_1P_EMAIL="$QA_1P_EMAIL" --env QA_EE_ACTIVATION_CODE="$QA_EE_ACTIVATION_CODE" --env QA_GOOGLE_CLIENT_EMAIL="$QA_GOOGLE_CLIENT_EMAIL" --env QA_ALLURE_AWS_SECRET_ACCESS_KEY="$QA_ALLURE_AWS_SECRET_ACCESS_KEY" --env QA_GCS_BUCKET_NAME="$QA_GCS_BUCKET_NAME" --env QA_GOOGLE_PROJECT="$QA_GOOGLE_PROJECT" --env QA_KNAPSACK_REPORT_GCS_CREDENTIALS="$QA_KNAPSACK_REPORT_GCS_CREDENTIALS" --env QA_AWS_S3_KEY_ID="$QA_AWS_S3_KEY_ID" --env QA_RUN_ALL_TESTS="$QA_RUN_ALL_TESTS" --env QA_GITLAB_CI_TOKEN="$QA_GITLAB_CI_TOKEN" --env QA_FRAMEWORK_CHANGES="$QA_FRAMEWORK_CHANGES" --env QA_INTERCEPT_REQUESTS="$QA_INTERCEPT_REQUESTS" --env QA_INFLUXDB_URL="$QA_INFLUXDB_URL" --env QA_THIRD_PARTY_DOCKER_USER="$QA_THIRD_PARTY_DOCKER_USER" --env QA_THIRD_PARTY_DOCKER_PASSWORD="$QA_THIRD_PARTY_DOCKER_PASSWORD" --env QA_1P_SECRET="$QA_1P_SECRET" --env QA_AWS_S3_ACCESS_KEY="$QA_AWS_S3_ACCESS_KEY" --env QA_LOG_LEVEL="$QA_LOG_LEVEL" --env QA_GITHUB_OAUTH_APP_ID="$QA_GITHUB_OAUTH_APP_ID" --env QA_EXPORT_TEST_METRICS="$QA_EXPORT_TEST_METRICS" --env QA_GOOGLE_CDN_LB="$QA_GOOGLE_CDN_LB" --env QA_1P_GITHUB_UUID="$QA_1P_GITHUB_UUID" --env QA_GCS_CDN_BUCKET_NAME="$QA_GCS_CDN_BUCKET_NAME" --env QA_SCENARIO="$QA_SCENARIO" --env QA_GOOGLE_JSON_KEY="$QA_GOOGLE_JSON_KEY" --env QA_SUITES="$QA_SUITES" --env QA_RUN_ALL_E2E_LABEL="$QA_RUN_ALL_E2E_LABEL" --env QA_ALLURE_AWS_ACCESS_KEY_ID="$QA_ALLURE_AWS_ACCESS_KEY_ID" --env QA_AWS_S3_BUCKET_NAME="$QA_AWS_S3_BUCKET_NAME" --env QA_GENERATE_ALLURE_REPORT="$QA_GENERATE_ALLURE_REPORT" --env QA_CAN_TEST_PRAEFECT="$QA_CAN_TEST_PRAEFECT" --env QA_RUN_TESTS_ON_GDK="$QA_RUN_TESTS_ON_GDK" --env QA_INFLUXDB_TOKEN="$QA_INFLUXDB_TOKEN" --env QA_TEST_LICENSE_MODE="$QA_TEST_LICENSE_MODE" --env QA_TEST_SESSION_TOKEN="$QA_TEST_SESSION_TOKEN" --env QA_GITHUB_ACCESS_TOKEN="$QA_GITHUB_ACCESS_TOKEN" --env QA_THIRD_PARTY_DOCKER_REPOSITORY="$QA_THIRD_PARTY_DOCKER_REPOSITORY" --env GITLAB_QA_ADMIN_ACCESS_TOKEN="$GITLAB_QA_ADMIN_ACCESS_TOKEN" --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" --env EE_LICENSE="$EE_LICENSE" --env GITLAB_QA_PASSWORD_1="$GITLAB_QA_PASSWORD_1" --env GITLAB_QA_PASSWORD_2="$GITLAB_QA_PASSWORD_2" --env CI="$CI" --env CI_COMMIT_REF_NAME="$CI_COMMIT_REF_NAME" --env CI_JOB_NAME="$CI_JOB_NAME" --env CI_JOB_URL="$CI_JOB_URL" --env CI_JOB_TOKEN="$CI_JOB_TOKEN" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env CI_PROJECT_ID="$CI_PROJECT_ID" --env CI_PROJECT_NAME="$CI_PROJECT_NAME" --env CI_PROJECT_PATH="$CI_PROJECT_PATH" --env CI_SLACK_WEBHOOK_URL="$CI_SLACK_WEBHOOK_URL" --env CI_PIPELINE_ID="$CI_PIPELINE_ID" --env CI_PIPELINE_SOURCE="$CI_PIPELINE_SOURCE" --env CI_PIPELINE_URL="$CI_PIPELINE_URL" --env CI_PIPELINE_CREATED_AT="$CI_PIPELINE_CREATED_AT" --env CI_MERGE_REQUEST_IID="$CI_MERGE_REQUEST_IID" --env CI_API_V4_URL="$CI_API_V4_URL" --env GITLAB_CI="$GITLAB_CI" --env GITLAB_QA_USER_AGENT="$GITLAB_QA_USER_AGENT" --env GITLAB_INITIAL_ROOT_PASSWORD="5iveL!fe" --env COLORIZED_LOGS="$COLORIZED_LOGS" --env RELEASE="$RELEASE" --env RELEASE_REGISTRY_URL="registry.gitlab.com" --env RELEASE_REGISTRY_USERNAME="gitlab-ci-token" --env RELEASE_REGISTRY_PASSWORD="[MASKED]" --env SELENOID_DIRECTORY="/builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/selenoid" --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-05-05-00-43-11-e47000e8/gitlab-ee-qa-fb5140ab:/home/gitlab/qa/tmp:z --volume /var/run/docker.sock:/var/run/docker.sock:z --name gitlab-ee-qa-fb5140ab registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:53f90aa57f8af288454aecc48b4a67b4ba12f638 Test::Instance::All http://gitlab-gitaly-cluster.test -- --format QA::Support::JsonFormatter --out tmp/rspec-4230518628.json --format RspecJunitFormatter --out tmp/rspec-4230518628.xml --format html --out tmp/rspec-gitaly-cluster.htm --color --format documentation --tag gitaly_cluster`326warning: parser/current is loading parser/ruby30, which recognizes3.0.5-compliant syntax, but you are running 3.0.6.3282023-05-05 00:49:47 WARN Selenium [:logger_info] Details on how to use and modify Selenium logger:3302023-05-05 00:49:47 WARN Selenium [DEPRECATION] [:caps_browsers] Remote::Capabilities.chrome is deprecated. Use Options.chrome instead.3312023-05-05 00:49:47 WARN Selenium [DEPRECATION] [:capabilities] The :capabilities parameter for Selenium::WebDriver::Chrome::Driver is deprecated. Use :options argument with an instance of Selenium::WebDriver::Chrome::Driver instead.3322023-05-05 00:49:47 / CONF :: 333 ==> Base URL: http://gitlab-gitaly-cluster.test334 ==> Browser: #<Selenium::WebDriver::Chrome::Driver:0x00007fdfb834c200>335 ==> Libraries: Chemlab::Vendor336[May 05 2023 00:49:47 UTC (QA Tests)] INFO -- Caching token for username: root, last six chars of token:sixwPA337[May 05 2023 00:49:47 UTC (QA Tests)] INFO -- Browser: chrome338[May 05 2023 00:49:47 UTC (QA Tests)] INFO -- Browser version: latest339[May 05 2023 00:49:47 UTC (QA Tests)] INFO -- Performing sanity check for environment!340[May 05 2023 00:49:48 UTC (QA Tests)] INFO -- Performing initial license fabrication!341[May 05 2023 00:49:48 UTC (QA Tests)] INFO -- Successfully added license key. Details:342{:test_license_mode=>false, :license_key_length=>2136, :last_five_characters=>"cbiJ9", :plan=>"ultimate", :starts_at=>"2023-02-03", :user_limit=>10000}343[May 05 2023 00:49:48 UTC (QA Tests)] INFO -- ==> Built a QA::EE::Resource::License with id '1' via api in 0.31 seconds344Run options:345 include {:gitaly_cluster=>true}346 exclude {:geo=>true, :requires_praefect=>true}347Randomized with seed 58080348Systems349 Gitaly distributed reads350[May 05 2023 00:52:05 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly distributed reads reads from each node351[May 05 2023 00:52:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`352[May 05 2023 00:52:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""353`354[May 05 2023 00:52:06 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`355[May 05 2023 00:52:06 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`356[May 05 2023 00:52:06 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`357[May 05 2023 00:52:06 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`358[May 05 2023 00:52:06 UTC (QA Tests)] INFO -- Waiting for health check on praefect359[May 05 2023 00:52:06 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`360[May 05 2023 00:52:07 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1361[May 05 2023 00:52:07 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`362[May 05 2023 00:52:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""363`364[May 05 2023 00:52:09 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2365[May 05 2023 00:52:09 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`366[May 05 2023 00:52:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""367`368[May 05 2023 00:52:10 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3369[May 05 2023 00:52:10 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`370[May 05 2023 00:52:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""371`372[May 05 2023 00:52:12 UTC (QA Tests)] INFO -- Retrieved cached token for username: root, last six chars of token:sixwPA373[May 05 2023 00:52:12 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::PersonalAccessToken with name 'api-pat-root-l6kyri77' via cache in 0.0 seconds374[May 05 2023 00:52:13 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.95 seconds375[May 05 2023 00:52:14 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 2.2 seconds376[May 05 2023 00:52:16 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-92ac8610bccbdb96' via api in 4.76 seconds377[May 05 2023 00:52:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""378`379[May 05 2023 00:52:17 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[2]"'`380[May 05 2023 00:52:44 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`381[May 05 2023 00:52:45 UTC (QA Tests)] INFO -- Reading from the repository382[May 05 2023 00:52:46 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`383[May 05 2023 00:52:52 UTC (QA Tests)] INFO -- Reading from the repository384[May 05 2023 00:52:53 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`385[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`386 reads from each node387 when a node is unhealthy388[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly distributed reads when a node is unhealthy does not read from the unhealthy node389[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`390[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""391`392[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`393[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`394[May 05 2023 00:52:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`395[May 05 2023 00:52:55 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`396[May 05 2023 00:52:55 UTC (QA Tests)] INFO -- Waiting for health check on praefect397[May 05 2023 00:52:55 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`398[May 05 2023 00:52:56 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1399[May 05 2023 00:52:56 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`400[May 05 2023 00:52:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""401`402[May 05 2023 00:52:57 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2403[May 05 2023 00:52:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`404[May 05 2023 00:52:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""405`406[May 05 2023 00:52:59 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3407[May 05 2023 00:52:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`408[May 05 2023 00:53:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""409`410[May 05 2023 00:53:00 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.08 seconds411[May 05 2023 00:53:00 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.18 seconds412[May 05 2023 00:53:03 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-c79ceef667700e43' via api in 2.43 seconds413[May 05 2023 00:53:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""414`415[May 05 2023 00:53:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[3]"'`416[May 05 2023 00:53:31 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`417[May 05 2023 00:53:31 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`418[May 05 2023 00:53:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""419`420[May 05 2023 00:53:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""421`422[May 05 2023 00:53:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""423`424[May 05 2023 00:53:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""425`426[May 05 2023 00:53:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""427`428[May 05 2023 00:53:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""429`430[May 05 2023 00:53:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""431`432[May 05 2023 00:53:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""433`434[May 05 2023 00:53:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""435`436[May 05 2023 00:53:41 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`437[May 05 2023 00:53:41 UTC (QA Tests)] INFO -- Reading from the repository438[May 05 2023 00:53:49 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`439[May 05 2023 00:53:50 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`440[May 05 2023 00:53:51 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`441[May 05 2023 00:53:52 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`442[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`443[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`444[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`445[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`446[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`447 does not read from the unhealthy node448Systems449 Praefect connectivity commands450 in an unhealthy environment451[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Starting test: Systems Praefect connectivity commands in an unhealthy environment diagnoses connection issues to gitaly nodes452[May 05 2023 00:53:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`453[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""454`455[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`456[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`457[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`458[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`459[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Waiting for health check on praefect460[May 05 2023 00:53:55 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`461[May 05 2023 00:53:57 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1462[May 05 2023 00:53:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`463[May 05 2023 00:53:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""464`465[May 05 2023 00:53:58 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2466[May 05 2023 00:53:58 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`467[May 05 2023 00:53:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""468`469[May 05 2023 00:53:59 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3470[May 05 2023 00:53:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`471[May 05 2023 00:54:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""472`473[May 05 2023 00:54:01 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`474[May 05 2023 00:54:01 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`475[May 05 2023 00:54:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""476`477[May 05 2023 00:54:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""478`479[May 05 2023 00:54:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""480`481[May 05 2023 00:54:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""482`483[May 05 2023 00:54:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""484`485[May 05 2023 00:54:07 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""486`487[May 05 2023 00:54:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""488`489[May 05 2023 00:54:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""490`491[May 05 2023 00:54:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""492`493[May 05 2023 00:54:11 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`494[May 05 2023 00:54:11 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`495[May 05 2023 00:54:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""496`497[May 05 2023 00:54:13 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""498`499[May 05 2023 00:54:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""500`501[May 05 2023 00:54:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""502`503[May 05 2023 00:54:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""504`505[May 05 2023 00:54:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""506`507[May 05 2023 00:54:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""508`509[May 05 2023 00:54:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""510`511[May 05 2023 00:54:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""512`513[May 05 2023 00:54:22 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`514[May 05 2023 00:54:23 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`515[May 05 2023 00:54:25 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`516[May 05 2023 00:54:26 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`517[May 05 2023 00:54:26 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`518[May 05 2023 00:54:27 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""519`520[May 05 2023 00:54:28 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""521`522[May 05 2023 00:54:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""523`524[May 05 2023 00:54:30 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""525`526[May 05 2023 00:54:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""527`528[May 05 2023 00:54:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""529`530[May 05 2023 00:54:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""531`532[May 05 2023 00:54:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""533`534[May 05 2023 00:54:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""535`536[May 05 2023 00:54:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""537`538[May 05 2023 00:54:38 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`539 diagnoses connection issues to gitaly nodes540[May 05 2023 00:54:39 UTC (QA Tests)] INFO -- Starting test: Systems Praefect connectivity commands in an unhealthy environment diagnoses unhealthy connection to database541[May 05 2023 00:54:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`542[May 05 2023 00:54:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""543`544[May 05 2023 00:54:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`545[May 05 2023 00:54:40 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`546[May 05 2023 00:54:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`547[May 05 2023 00:54:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`548[May 05 2023 00:54:40 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`549[May 05 2023 00:54:41 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`550[May 05 2023 00:54:41 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`551[May 05 2023 00:54:41 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`552[May 05 2023 00:54:41 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`553[May 05 2023 00:54:41 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`554[May 05 2023 00:54:42 UTC (QA Tests)] INFO -- Waiting for health check on praefect555[May 05 2023 00:54:42 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`556[May 05 2023 00:54:43 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1557[May 05 2023 00:54:43 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`558[May 05 2023 00:54:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""559`560[May 05 2023 00:54:44 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2561[May 05 2023 00:54:44 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`562[May 05 2023 00:54:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""563`564[May 05 2023 00:54:46 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3565[May 05 2023 00:54:46 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`566[May 05 2023 00:54:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""567`568[May 05 2023 00:54:47 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`569[May 05 2023 00:54:47 UTC (QA Tests)] INFO -- Executing: `docker pause postgres`570[May 05 2023 00:54:48 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping'`571 diagnoses unhealthy connection to database572 in a healthy environment573[May 05 2023 00:55:18 UTC (QA Tests)] INFO -- Starting test: Systems Praefect connectivity commands in a healthy environment confirms healthy connection to gitaly nodes574[May 05 2023 00:55:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`575[May 05 2023 00:55:18 UTC (QA Tests)] INFO -- Executing: `docker unpause postgres`576[May 05 2023 00:55:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} postgres`577[May 05 2023 00:55:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""578`579[May 05 2023 00:55:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`580[May 05 2023 00:55:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`581[May 05 2023 00:55:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`582[May 05 2023 00:55:20 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`583[May 05 2023 00:55:20 UTC (QA Tests)] INFO -- Waiting for health check on praefect584[May 05 2023 00:55:20 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`585[May 05 2023 00:55:21 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1586[May 05 2023 00:55:21 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`587[May 05 2023 00:55:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""588`589[May 05 2023 00:55:23 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2590[May 05 2023 00:55:23 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`591[May 05 2023 00:55:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""592`593[May 05 2023 00:55:24 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3594[May 05 2023 00:55:24 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`595[May 05 2023 00:55:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""596`597[May 05 2023 00:55:26 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`598[May 05 2023 00:55:26 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`599[May 05 2023 00:55:27 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes -timeout 1s'`600 confirms healthy connection to gitaly nodes601[May 05 2023 00:55:27 UTC (QA Tests)] INFO -- Starting test: Systems Praefect connectivity commands in a healthy environment confirms healthy connection to database602[May 05 2023 00:55:27 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`603[May 05 2023 00:55:28 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""604`605[May 05 2023 00:55:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`606[May 05 2023 00:55:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`607[May 05 2023 00:55:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`608[May 05 2023 00:55:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`609[May 05 2023 00:55:29 UTC (QA Tests)] INFO -- Waiting for health check on praefect610[May 05 2023 00:55:29 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`611[May 05 2023 00:55:30 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1612[May 05 2023 00:55:30 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`613[May 05 2023 00:55:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""614`615[May 05 2023 00:55:32 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2616[May 05 2023 00:55:32 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`617[May 05 2023 00:55:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""618`619[May 05 2023 00:55:33 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3620[May 05 2023 00:55:33 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`621[May 05 2023 00:55:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""622`623[May 05 2023 00:55:35 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping'`624 confirms healthy connection to database625Systems626 Praefect repository commands627[May 05 2023 00:55:36 UTC (QA Tests)] INFO -- Starting test: Systems Praefect repository commands allows admin to manage difference between praefect database and disk state628[May 05 2023 00:55:36 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`629[May 05 2023 00:55:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""630`631[May 05 2023 00:55:36 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`632[May 05 2023 00:55:36 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`633[May 05 2023 00:55:37 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`634[May 05 2023 00:55:37 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`635[May 05 2023 00:55:37 UTC (QA Tests)] INFO -- Waiting for health check on praefect636[May 05 2023 00:55:37 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`637[May 05 2023 00:55:38 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1638[May 05 2023 00:55:38 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`639[May 05 2023 00:55:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""640`641[May 05 2023 00:55:40 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2642[May 05 2023 00:55:40 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`643[May 05 2023 00:55:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""644`645[May 05 2023 00:55:42 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3646[May 05 2023 00:55:42 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`647[May 05 2023 00:55:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""648`649[May 05 2023 00:55:43 UTC (QA Tests)] INFO -- Executing: `docker exec --user git gitaly1 bash -c 'GIT_DIR=. git init --initial-branch=main /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`650[May 05 2023 00:55:43 UTC (QA Tests)] INFO -- Executing: `651 docker exec --user git gitaly1 bash -c 'find /var/opt/gitlab/git-data/repositories/@hashed/repo1.git -exec touch -d "24 hours ago" {} \;'652 `653[May 05 2023 00:55:44 UTC (QA Tests)] INFO -- Executing: `docker exec --user git gitaly3 bash -c 'GIT_DIR=. git init --initial-branch=main /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`654[May 05 2023 00:55:44 UTC (QA Tests)] INFO -- Executing: `655 docker exec --user git gitaly3 bash -c 'find /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git -exec touch -d "24 hours ago" {} \;'656 `657[May 05 2023 00:55:44 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect list-untracked-repositories'`658[May 05 2023 00:55:46 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect track-repository --repository-relative-path @hashed/repo1.git --authoritative-storage gitaly1 --virtual-storage-name default'`659[May 05 2023 00:55:48 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect list-untracked-repositories'`660[May 05 2023 00:55:49 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/repo1.git || echo -n 'DIR_DOES_NOT_EXIST''`661[May 05 2023 00:55:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/repo1.git';\""662`663[May 05 2023 00:55:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM repositories where relative_path='@hashed/repo1.git';\""664`665[May 05 2023 00:55:50 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect track-repository --repository-relative-path @hashed/path/to/repo2.git --authoritative-storage gitaly3 --virtual-storage-name default'`666[May 05 2023 00:55:52 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect list-untracked-repositories'`667[May 05 2023 00:55:54 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git || echo -n 'DIR_DOES_NOT_EXIST''`668[May 05 2023 00:55:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/path/to/repo2.git';\""669`670[May 05 2023 00:55:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM repositories where relative_path='@hashed/path/to/repo2.git';\""671`672[May 05 2023 00:55:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/repo1.git';\""673`674[May 05 2023 00:55:55 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/repo1.git || echo -n 'DIR_DOES_NOT_EXIST''`675[May 05 2023 00:55:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/repo1.git';\""676`677[May 05 2023 00:55:55 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/repo1.git || echo -n 'DIR_DOES_NOT_EXIST''`678[May 05 2023 00:55:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/path/to/repo2.git';\""679`680[May 05 2023 00:55:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/path/to/repo2.git';\""681`682[May 05 2023 00:55:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git || echo -n 'DIR_DOES_NOT_EXIST''`683[May 05 2023 00:55:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/path/to/repo2.git';\""684`685[May 05 2023 00:55:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git || echo -n 'DIR_DOES_NOT_EXIST''`686[May 05 2023 00:55:58 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect remove-repository --repository-relative-path @hashed/repo1.git --virtual-storage-name default --apply'`687[May 05 2023 00:55:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/repo1.git || echo -n 'DIR_DOES_NOT_EXIST''`688[May 05 2023 00:55:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/repo1.git || echo -n 'DIR_DOES_NOT_EXIST''`689[May 05 2023 00:56:00 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'test -d /var/opt/gitlab/git-data/repositories/@hashed/repo1.git || echo -n 'DIR_DOES_NOT_EXIST''`690[May 05 2023 00:56:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM storage_repositories where relative_path='@hashed/repo1.git';\""691`692[May 05 2023 00:56:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM repositories where relative_path='@hashed/repo1.git';\""693`694[May 05 2023 00:56:00 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect list-untracked-repositories'`695[May 05 2023 00:56:02 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`696[May 05 2023 00:56:02 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`697[May 05 2023 00:56:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`698[May 05 2023 00:56:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`699[May 05 2023 00:56:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`700[May 05 2023 00:56:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`701[May 05 2023 00:56:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from repositories where relative_path = '@hashed/repo1.git';\""702`703[May 05 2023 00:56:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from storage_repositories where relative_path = '@hashed/repo1.git';\""704`705[May 05 2023 00:56:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from repositories where relative_path = '@hashed/path/to/repo2.git';\""706`707[May 05 2023 00:56:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from storage_repositories where relative_path = '@hashed/path/to/repo2.git';\""708`709 allows admin to manage difference between praefect database and disk state710[May 05 2023 00:56:05 UTC (QA Tests)] INFO -- Starting test: Systems Praefect repository commands allows admin to control the number of replicas of data711[May 05 2023 00:56:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`712[May 05 2023 00:56:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""713`714[May 05 2023 00:56:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`715[May 05 2023 00:56:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`716[May 05 2023 00:56:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`717[May 05 2023 00:56:06 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`718[May 05 2023 00:56:06 UTC (QA Tests)] INFO -- Waiting for health check on praefect719[May 05 2023 00:56:06 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`720[May 05 2023 00:56:07 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1721[May 05 2023 00:56:07 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`722[May 05 2023 00:56:08 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""723`724[May 05 2023 00:56:09 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2725[May 05 2023 00:56:09 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`726[May 05 2023 00:56:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""727`728[May 05 2023 00:56:10 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3729[May 05 2023 00:56:10 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`730[May 05 2023 00:56:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""731`732[May 05 2023 00:56:12 UTC (QA Tests)] INFO -- Executing: `docker exec --user git gitaly1 bash -c 'GIT_DIR=. git init --initial-branch=main /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`733[May 05 2023 00:56:12 UTC (QA Tests)] INFO -- Executing: `734 docker exec --user git gitaly1 bash -c 'find /var/opt/gitlab/git-data/repositories/@hashed/repo1.git -exec touch -d "24 hours ago" {} \;'735 `736[May 05 2023 00:56:12 UTC (QA Tests)] INFO -- Executing: `docker exec --user git gitaly3 bash -c 'GIT_DIR=. git init --initial-branch=main /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`737[May 05 2023 00:56:12 UTC (QA Tests)] INFO -- Executing: `738 docker exec --user git gitaly3 bash -c 'find /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git -exec touch -d "24 hours ago" {} \;'739 `740[May 05 2023 00:56:13 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c 'gitlab-ctl praefect track-repository --repository-relative-path @hashed/repo1.git --authoritative-storage gitaly1 --virtual-storage-name default'`741[May 05 2023 00:56:14 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -repository @hashed/repo1.git -virtual-storage default -replication-factor 2'`742[May 05 2023 00:56:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT storage FROM repository_assignments WHERE relative_path='@hashed/repo1.git' AND virtual_storage='default';\""743`744[May 05 2023 00:56:15 UTC (QA Tests)] INFO -- Executing: `docker exec praefect bash -c '/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -repository @hashed/repo1.git -virtual-storage default -replication-factor 3'`745[May 05 2023 00:56:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT storage FROM repository_assignments WHERE relative_path='@hashed/repo1.git' AND virtual_storage='default';\""746`747[May 05 2023 00:56:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`748[May 05 2023 00:56:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`749[May 05 2023 00:56:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/repo1.git'`750[May 05 2023 00:56:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`751[May 05 2023 00:56:17 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`752[May 05 2023 00:56:17 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 bash -c 'rm -rf /var/opt/gitlab/git-data/repositories/@hashed/path/to/repo2.git'`753[May 05 2023 00:56:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from repositories where relative_path = '@hashed/repo1.git';\""754`755[May 05 2023 00:56:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from storage_repositories where relative_path = '@hashed/repo1.git';\""756`757[May 05 2023 00:56:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from repositories where relative_path = '@hashed/path/to/repo2.git';\""758`759[May 05 2023 00:56:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from storage_repositories where relative_path = '@hashed/path/to/repo2.git';\""760`761 allows admin to control the number of replicas of data762Systems763 Praefect dataloss commands764[May 05 2023 00:56:18 UTC (QA Tests)] INFO -- Starting test: Systems Praefect dataloss commands confirms that changes are synced across all storages765[May 05 2023 00:56:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`766[May 05 2023 00:56:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""767`768[May 05 2023 00:56:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`769[May 05 2023 00:56:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`770[May 05 2023 00:56:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`771[May 05 2023 00:56:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`772[May 05 2023 00:56:19 UTC (QA Tests)] INFO -- Waiting for health check on praefect773[May 05 2023 00:56:19 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`774[May 05 2023 00:56:21 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1775[May 05 2023 00:56:21 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`776[May 05 2023 00:56:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""777`778[May 05 2023 00:56:22 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2779[May 05 2023 00:56:22 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`780[May 05 2023 00:56:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""781`782[May 05 2023 00:56:24 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3783[May 05 2023 00:56:24 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`784[May 05 2023 00:56:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""785`786[May 05 2023 00:56:25 UTC (QA Tests)] INFO -- Running eventually matcher with 'include' operator with: '{:max_duration=>60, :sleep_interval=>0.5}' arguments787[May 05 2023 00:56:25 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.09 seconds788[May 05 2023 00:56:25 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.22 seconds789[May 05 2023 00:56:28 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-dataloss-project-8b02c23ab6075ffd' via api in 2.84 seconds790[May 05 2023 00:56:28 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`791 confirms that changes are synced across all storages792[May 05 2023 00:56:28 UTC (QA Tests)] INFO -- Starting test: Systems Praefect dataloss commands identifies how many changes are not in sync across storages793[May 05 2023 00:56:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`794[May 05 2023 00:56:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""795`796[May 05 2023 00:56:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`797[May 05 2023 00:56:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`798[May 05 2023 00:56:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`799[May 05 2023 00:56:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`800[May 05 2023 00:56:30 UTC (QA Tests)] INFO -- Waiting for health check on praefect801[May 05 2023 00:56:30 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`802[May 05 2023 00:56:31 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1803[May 05 2023 00:56:31 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`804[May 05 2023 00:56:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""805`806[May 05 2023 00:56:32 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2807[May 05 2023 00:56:32 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`808[May 05 2023 00:56:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""809`810[May 05 2023 00:56:34 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3811[May 05 2023 00:56:34 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`812[May 05 2023 00:56:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""813`814[May 05 2023 00:56:35 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.06 seconds815[May 05 2023 00:56:35 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.24 seconds816[May 05 2023 00:56:39 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-dataloss-project-6de1a0e10c349e57' via api in 3.53 seconds817[May 05 2023 00:56:39 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`818[May 05 2023 00:56:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`819[May 05 2023 00:56:39 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`820[May 05 2023 00:56:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""821`822[May 05 2023 00:56:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""823`824[May 05 2023 00:56:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""825`826[May 05 2023 00:56:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""827`828[May 05 2023 00:56:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""829`830[May 05 2023 00:56:46 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""831`832[May 05 2023 00:56:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""833`834[May 05 2023 00:56:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""835`836[May 05 2023 00:56:49 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""837`838[May 05 2023 00:56:51 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.49 seconds839[May 05 2023 00:56:51 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.43 seconds840[May 05 2023 00:56:53 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.76 seconds841[May 05 2023 00:56:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`842[May 05 2023 00:56:53 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`843[May 05 2023 00:56:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""844`845[May 05 2023 00:56:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""846`847[May 05 2023 00:56:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""848`849[May 05 2023 00:56:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""850`851[May 05 2023 00:56:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""852`853[May 05 2023 00:56:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""854`855[May 05 2023 00:57:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""856`857[May 05 2023 00:57:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""858`859[May 05 2023 00:57:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""860`861[May 05 2023 00:57:03 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`862 identifies how many changes are not in sync across storages863[May 05 2023 00:57:04 UTC (QA Tests)] INFO -- Starting test: Systems Praefect dataloss commands allows admin resolve scenario where data cannot be recovered864[May 05 2023 00:57:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`865[May 05 2023 00:57:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""866`867[May 05 2023 00:57:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`868[May 05 2023 00:57:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`869[May 05 2023 00:57:04 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`870[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`871[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`872[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`873[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`874[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`875[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Waiting for health check on praefect876[May 05 2023 00:57:05 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`877[May 05 2023 00:57:07 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1878[May 05 2023 00:57:07 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`879[May 05 2023 00:57:08 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""880`881[May 05 2023 00:57:08 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2882[May 05 2023 00:57:08 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`883[May 05 2023 00:57:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""884`885[May 05 2023 00:57:09 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3886[May 05 2023 00:57:09 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`887[May 05 2023 00:57:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""888`889[May 05 2023 00:57:11 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.06 seconds890[May 05 2023 00:57:11 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.39 seconds891[May 05 2023 00:57:12 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-dataloss-project-80cd8c5d2f6b3d36' via api in 1.45 seconds892[May 05 2023 00:57:12 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`893[May 05 2023 00:57:15 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 2.32 seconds894[May 05 2023 00:57:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"895 select * from replication_queue896 where state = 'ready'897 and job ->> 'change' = 'update'898 and job ->> 'target_node_storage' = 'gitaly1'899 and job ->> 'relative_path' = 'e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git';900 \""901`902[May 05 2023 00:57:15 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`903[May 05 2023 00:57:15 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`904[May 05 2023 00:57:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""905`906[May 05 2023 00:57:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""907`908[May 05 2023 00:57:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""909`910[May 05 2023 00:57:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""911`912[May 05 2023 00:57:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""913`914[May 05 2023 00:57:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""915`916[May 05 2023 00:57:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""917`918[May 05 2023 00:57:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""919`920[May 05 2023 00:57:26 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""921`922[May 05 2023 00:57:27 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 1.2 seconds923[May 05 2023 00:57:27 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"924 select * from replication_queue925 where state = 'ready'926 and job ->> 'change' = 'update'927 and job ->> 'target_node_storage' = 'gitaly2'928 and job ->> 'relative_path' = 'e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git';929 \""930`931[May 05 2023 00:57:27 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`932[May 05 2023 00:57:27 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`933[May 05 2023 00:57:28 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""934`935[May 05 2023 00:57:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""936`937[May 05 2023 00:57:30 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""938`939[May 05 2023 00:57:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""940`941[May 05 2023 00:57:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""942`943[May 05 2023 00:57:34 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""944`945[May 05 2023 00:57:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""946`947[May 05 2023 00:57:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""948`949[May 05 2023 00:57:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""950`951[May 05 2023 00:57:39 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.84 seconds952[May 05 2023 00:57:39 UTC (QA Tests)] INFO -- Executing: `953 docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss --virtual-storage=default --repository=@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git --authoritative-storage=gitaly2954 `955[May 05 2023 00:57:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`956[May 05 2023 00:57:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""957`958[May 05 2023 00:57:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`959[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`960[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`961[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`962[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`963[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`964[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`965[May 05 2023 00:57:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`966[May 05 2023 00:57:41 UTC (QA Tests)] INFO -- Waiting for health check on praefect967[May 05 2023 00:57:41 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`968[May 05 2023 00:57:42 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1969[May 05 2023 00:57:42 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`970[May 05 2023 00:57:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""971`972[May 05 2023 00:57:43 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2973[May 05 2023 00:57:43 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`974[May 05 2023 00:57:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""975`976[May 05 2023 00:57:45 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3977[May 05 2023 00:57:45 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`978[May 05 2023 00:57:46 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""979`980[May 05 2023 00:57:46 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`981[May 05 2023 00:57:46 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`982[May 05 2023 00:57:47 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`983[May 05 2023 00:57:47 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`984[May 05 2023 00:57:48 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`985[May 05 2023 00:57:48 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`986[May 05 2023 00:57:49 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`987[May 05 2023 00:57:49 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`988[May 05 2023 00:57:50 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`989[May 05 2023 00:57:50 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`990[May 05 2023 00:57:51 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`991[May 05 2023 00:57:51 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`992[May 05 2023 00:57:51 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`993[May 05 2023 00:57:52 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`994[May 05 2023 00:57:52 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`995[May 05 2023 00:57:53 UTC (QA Tests)] INFO -- Executing: `docker exec praefect praefect -config /var/opt/gitlab/praefect/config.toml dataloss --partially-unavailable=true`996 allows admin resolve scenario where data cannot be recovered997Systems998 Gitaly Cluster replication queue999[May 05 2023 00:57:53 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly Cluster replication queue allows replication of different repository after interruption1000[May 05 2023 00:57:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1001[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1002`1003[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1004[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1005[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1006[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1007[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Waiting for health check on praefect1008[May 05 2023 00:57:54 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1009[May 05 2023 00:57:56 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11010[May 05 2023 00:57:56 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1011[May 05 2023 00:57:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1012`1013[May 05 2023 00:57:57 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21014[May 05 2023 00:57:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1015[May 05 2023 00:57:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1016`1017[May 05 2023 00:57:58 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31018[May 05 2023 00:57:58 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1019[May 05 2023 00:58:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1020`1021[May 05 2023 00:58:00 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1022[May 05 2023 00:58:00 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`1023[May 05 2023 00:58:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1024`1025[May 05 2023 00:58:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1026`1027[May 05 2023 00:58:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1028`1029[May 05 2023 00:58:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1030`1031[May 05 2023 00:58:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1032`1033[May 05 2023 00:58:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1034`1035[May 05 2023 00:58:07 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1036`1037[May 05 2023 00:58:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1038`1039[May 05 2023 00:58:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1040`1041[May 05 2023 00:58:10 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.07 seconds1042[May 05 2023 00:58:10 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.21 seconds1043[May 05 2023 00:58:11 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-d8e7a5d7639a5b03' via api in 1.16 seconds1044[May 05 2023 00:58:38 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1045[May 05 2023 00:58:38 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`1046[May 05 2023 00:58:38 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`1047[May 05 2023 00:58:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""1048`1049[May 05 2023 00:58:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""1050`1051[May 05 2023 00:58:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""1052`1053[May 05 2023 00:58:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""1054`1055[May 05 2023 00:58:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""1056`1057[May 05 2023 00:58:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1058[May 05 2023 00:58:40 UTC (QA Tests)] INFO -- Executing: `docker pause praefect`1059[May 05 2023 00:58:40 UTC (QA Tests)] INFO -- Setting jobs in replication queue to `in_progress` and acquiring locks1060[May 05 2023 00:58:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"update replication_queue set state = 'in_progress';1061insert into replication_queue_job_lock (job_id, lock_id, triggered_at)1062 select id, rq.lock_id, created_at from replication_queue rq1063 left join replication_queue_job_lock rqjl on rq.id = rqjl.job_id1064 where state = 'in_progress' and rqjl.job_id is null;1065update replication_queue_lock set acquired = 't';1066\""1067`1068[May 05 2023 00:58:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1069[May 05 2023 00:58:40 UTC (QA Tests)] INFO -- Executing: `docker unpause praefect`1070[May 05 2023 00:58:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} praefect`1071[May 05 2023 00:58:41 UTC (QA Tests)] INFO -- Waiting for health check on praefect1072[May 05 2023 00:58:41 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1073[May 05 2023 00:58:42 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.08 seconds1074[May 05 2023 00:58:42 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.36 seconds1075[May 05 2023 00:58:44 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/the_awesome_project-b70ff41478100b40' via api in 2.1 seconds1076[May 05 2023 00:58:44 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[8]"'`1077[May 05 2023 00:59:11 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1078[May 05 2023 00:59:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1079`1080[May 05 2023 00:59:12 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1081[May 05 2023 00:59:12 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1082[May 05 2023 00:59:12 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1083[May 05 2023 00:59:12 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1084[May 05 2023 00:59:12 UTC (QA Tests)] INFO -- Waiting for health check on praefect1085[May 05 2023 00:59:12 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1086[May 05 2023 00:59:13 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11087[May 05 2023 00:59:13 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1088[May 05 2023 00:59:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1089`1090[May 05 2023 00:59:15 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21091[May 05 2023 00:59:15 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1092[May 05 2023 00:59:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1093`1094[May 05 2023 00:59:16 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31095[May 05 2023 00:59:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1096[May 05 2023 00:59:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1097`1098[May 05 2023 00:59:17 UTC (QA Tests)] INFO -- Clearing the replication queue1099[May 05 2023 00:59:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from replication_queue_job_lock;1100delete from replication_queue_lock;1101delete from replication_queue;1102\""1103`1104 allows replication of different repository after interruption1105Systems1106 with Gitaly automatic failover and recovery1107[May 05 2023 00:59:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1108[May 05 2023 00:59:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1109`1110[May 05 2023 00:59:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1111[May 05 2023 00:59:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1112[May 05 2023 00:59:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1113[May 05 2023 00:59:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1114[May 05 2023 00:59:19 UTC (QA Tests)] INFO -- Waiting for health check on praefect1115[May 05 2023 00:59:19 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1116[May 05 2023 00:59:20 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11117[May 05 2023 00:59:20 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1118[May 05 2023 00:59:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1119`1120[May 05 2023 00:59:21 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21121[May 05 2023 00:59:21 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1122[May 05 2023 00:59:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1123`1124[May 05 2023 00:59:23 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31125[May 05 2023 00:59:23 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1126[May 05 2023 00:59:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1127`1128[May 05 2023 00:59:24 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.06 seconds1129[May 05 2023 00:59:24 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.18 seconds1130[May 05 2023 00:59:25 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-4b519e5eeb43039e' via api in 1.23 seconds1131[May 05 2023 00:59:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""1132`1133[May 05 2023 00:59:26 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[9]"'`1134[May 05 2023 00:59:53 UTC (QA Tests)] INFO -- Starting test: Systems with Gitaly automatic failover and recovery automatically fails over1135[May 05 2023 00:59:53 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1136[May 05 2023 00:59:53 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`1137[May 05 2023 00:59:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1138`1139[May 05 2023 00:59:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1140`1141[May 05 2023 00:59:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1142`1143[May 05 2023 00:59:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1144`1145[May 05 2023 00:59:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1146`1147[May 05 2023 00:59:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1148`1149[May 05 2023 01:00:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1150`1151[May 05 2023 01:00:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1152`1153[May 05 2023 01:00:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1154`1155[May 05 2023 01:00:03 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1156[May 05 2023 01:00:04 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`1157[May 05 2023 01:00:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1158`1159[May 05 2023 01:00:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1160`1161[May 05 2023 01:00:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1162`1163[May 05 2023 01:00:07 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1164`1165[May 05 2023 01:00:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1166`1167[May 05 2023 01:00:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1168`1169[May 05 2023 01:00:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1170`1171[May 05 2023 01:00:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1172`1173[May 05 2023 01:00:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1174`1175[May 05 2023 01:00:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1176`1177[May 05 2023 01:00:17 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "first_added_commit_message to primary gitaly node"1178[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.52 seconds1179[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1180[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1181`1182[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1183[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1184[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`1185[May 05 2023 01:00:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`1186[May 05 2023 01:00:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1187[May 05 2023 01:00:19 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`1188[May 05 2023 01:00:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`1189[May 05 2023 01:00:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1190[May 05 2023 01:00:19 UTC (QA Tests)] INFO -- Waiting for health check on praefect1191[May 05 2023 01:00:19 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1192[May 05 2023 01:00:20 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11193[May 05 2023 01:00:20 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1194[May 05 2023 01:00:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1195`1196[May 05 2023 01:00:22 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21197[May 05 2023 01:00:22 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1198[May 05 2023 01:00:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1199`1200[May 05 2023 01:00:23 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31201[May 05 2023 01:00:23 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1202[May 05 2023 01:00:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1203`1204[May 05 2023 01:00:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""1205`1206[May 05 2023 01:00:25 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[9]"'`1207[May 05 2023 01:00:52 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1208[May 05 2023 01:00:52 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`1209[May 05 2023 01:00:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1210`1211[May 05 2023 01:00:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1212`1213[May 05 2023 01:00:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1214`1215[May 05 2023 01:00:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1216`1217[May 05 2023 01:00:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1218`1219[May 05 2023 01:00:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1220`1221[May 05 2023 01:01:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1222`1223[May 05 2023 01:01:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1224`1225[May 05 2023 01:01:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1226`1227[May 05 2023 01:01:02 UTC (QA Tests)] INFO -- Waiting for health check failure on gitaly11228[May 05 2023 01:01:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1229`1230[May 05 2023 01:01:03 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.53 seconds1231 automatically fails over1232 when recovering from dataloss after failover1233[May 05 2023 01:01:03 UTC (QA Tests)] INFO -- Starting test: Systems with Gitaly automatic failover and recovery when recovering from dataloss after failover automatically reconciles1234[May 05 2023 01:01:03 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1235[May 05 2023 01:01:03 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`1236[May 05 2023 01:01:04 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`1237[May 05 2023 01:01:04 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11238[May 05 2023 01:01:04 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1239[May 05 2023 01:01:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1240`1241[May 05 2023 01:01:05 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[9]"'`1242[May 05 2023 01:01:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1243[May 05 2023 01:01:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1244`1245[May 05 2023 01:01:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1246[May 05 2023 01:01:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1247[May 05 2023 01:01:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1248[May 05 2023 01:01:34 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1249[May 05 2023 01:01:34 UTC (QA Tests)] INFO -- Waiting for health check on praefect1250[May 05 2023 01:01:34 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1251[May 05 2023 01:01:35 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11252[May 05 2023 01:01:35 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1253[May 05 2023 01:01:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1254`1255[May 05 2023 01:01:36 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21256[May 05 2023 01:01:36 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1257[May 05 2023 01:01:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1258`1259[May 05 2023 01:01:38 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31260[May 05 2023 01:01:38 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1261[May 05 2023 01:01:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1262`1263 automatically reconciles1264Systems1265 Gitaly backend node recovery1266[May 05 2023 01:01:39 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly backend node recovery recovers from dataloss1267[May 05 2023 01:01:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1268[May 05 2023 01:01:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1269`1270[May 05 2023 01:01:39 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1271[May 05 2023 01:01:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1272[May 05 2023 01:01:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1273[May 05 2023 01:01:40 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1274[May 05 2023 01:01:40 UTC (QA Tests)] INFO -- Waiting for health check on praefect1275[May 05 2023 01:01:40 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1276[May 05 2023 01:01:41 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11277[May 05 2023 01:01:41 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1278[May 05 2023 01:01:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1279`1280[May 05 2023 01:01:43 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21281[May 05 2023 01:01:43 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1282[May 05 2023 01:01:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1283`1284[May 05 2023 01:01:44 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31285[May 05 2023 01:01:44 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1286[May 05 2023 01:01:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1287`1288[May 05 2023 01:01:45 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.06 seconds1289[May 05 2023 01:01:46 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335' via api in 0.18 seconds1290[May 05 2023 01:01:47 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2023-05-05-00-52-12-49937cb21686a335/gitaly_cluster-dc1c949f8f32def9' via api in 1.33 seconds1291[May 05 2023 01:01:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""1292`1293[May 05 2023 01:01:47 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[10]"'`1294[May 05 2023 01:02:15 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1295[May 05 2023 01:02:15 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`1296[May 05 2023 01:02:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1297`1298[May 05 2023 01:02:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1299`1300[May 05 2023 01:02:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1301`1302[May 05 2023 01:02:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1303`1304[May 05 2023 01:02:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1305`1306[May 05 2023 01:02:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1307`1308[May 05 2023 01:02:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1309`1310[May 05 2023 01:02:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1311`1312[May 05 2023 01:02:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1313`1314[May 05 2023 01:02:25 UTC (QA Tests)] INFO -- Waiting for health check failure on gitaly11315[May 05 2023 01:02:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1316`1317[May 05 2023 01:02:27 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "pushed after failover"1318[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.38 seconds1319[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select job from replication_queue1320where state = 'ready'1321 and job ->> 'change' = 'update'1322 and job ->> 'target_node_storage' = 'gitaly1';1323\""1324`1325[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1326[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`1327[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`1328[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11329[May 05 2023 01:02:28 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1330[May 05 2023 01:02:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1331`1332[May 05 2023 01:02:30 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21333[May 05 2023 01:02:30 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1334[May 05 2023 01:02:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1335`1336[May 05 2023 01:02:31 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31337[May 05 2023 01:02:31 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1338[May 05 2023 01:02:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1339`1340[May 05 2023 01:02:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""1341`1342[May 05 2023 01:02:33 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[10]"'`1343[May 05 2023 01:03:00 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1344[May 05 2023 01:03:00 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`1345[May 05 2023 01:03:00 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1346`1347[May 05 2023 01:03:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1348`1349[May 05 2023 01:03:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1350`1351[May 05 2023 01:03:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1352`1353[May 05 2023 01:03:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1354`1355[May 05 2023 01:03:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1356`1357[May 05 2023 01:03:07 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1358`1359[May 05 2023 01:03:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1360`1361[May 05 2023 01:03:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1362`1363[May 05 2023 01:03:10 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1364[May 05 2023 01:03:10 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`1365[May 05 2023 01:03:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1366`1367[May 05 2023 01:03:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1368`1369[May 05 2023 01:03:13 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1370`1371[May 05 2023 01:03:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1372`1373[May 05 2023 01:03:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1374`1375[May 05 2023 01:03:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1376`1377[May 05 2023 01:03:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1378`1379[May 05 2023 01:03:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1380`1381[May 05 2023 01:03:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1382`1383[May 05 2023 01:03:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1384`1385[May 05 2023 01:03:22 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`1386[May 05 2023 01:03:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""1387`1388[May 05 2023 01:03:22 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`1389[May 05 2023 01:03:22 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`1390[May 05 2023 01:03:23 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`1391[May 05 2023 01:03:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`1392[May 05 2023 01:03:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`1393[May 05 2023 01:03:23 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`1394[May 05 2023 01:03:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`1395[May 05 2023 01:03:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`1396[May 05 2023 01:03:24 UTC (QA Tests)] INFO -- Waiting for health check on praefect1397[May 05 2023 01:03:24 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`1398[May 05 2023 01:03:25 UTC (QA Tests)] INFO -- Waiting for health check on gitaly11399[May 05 2023 01:03:25 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`1400[May 05 2023 01:03:26 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""1401`1402[May 05 2023 01:03:26 UTC (QA Tests)] INFO -- Waiting for health check on gitaly21403[May 05 2023 01:03:26 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`1404[May 05 2023 01:03:27 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""1405`1406[May 05 2023 01:03:27 UTC (QA Tests)] INFO -- Waiting for health check on gitaly31407[May 05 2023 01:03:27 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`1408[May 05 2023 01:03:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""1409`1410 recovers from dataloss1411Finished in 13 minutes 40 seconds (files took 3.94 seconds to load)141215 examples, 0 failures1413Randomized with seed 580801414[May 05 2023 01:03:30 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=gitlab-gitaly-cluster`1415[May 05 2023 01:03:30 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f gitlab-gitaly-cluster`1416[May 05 2023 01:03:30 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=telegraf`1417[May 05 2023 01:03:30 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f telegraf`1419Running after script...1420$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")1421$ echo -e "\e[0Ksection_start:`date +%s`:report_results_section[collapsed=true]\r\e[0KReport results"1429Not uploading cache qa-e2e-ruby-3.0-c9589dbb96af39a99372100e42ffffe8-16 due to policy1431Uploading artifacts...1432gitlab-qa-run-*: found 259 matching files and directories 1433Uploading artifacts as "archive" to coordinator... 201 Created id=4230518628 responseStatus=201 Created token=64_Ucp7b1434Uploading artifacts...1435gitlab-qa-run-*/**/rspec-*.xml: found 1 matching files and directories 1436Uploading artifacts as "junit" to coordinator... 201 Created id=4230518628 responseStatus=201 Created token=64_Ucp7b1438Job succeeded