QA failures when running with multiple unicorn/workhorse pods
We have been seeing regular qa failures when running 2 unicorn pods in: https://gitlab.com/charts/gitlab/merge_requests/285
The QA failures look similar to this:
1) cloning code using a deploy key user sets up a deploy key with QA::Runtime::Key::ECDSA(521) to clone code using pipelines
Failure/Error: expect(job.output).to include(sha1sum)
expected "Running with gitlab-runner 10.8.0 (079aad9e) on qa-runner-1527261495 18bc6999 Using Shell executor.....g: Permanently added the ECDSA host key for IP address '35.234.155.116' to the list of known hosts." to include "256d8c7a676a21a82b2c85e0ea22ba7c65976339"
# ./qa/specs/features/project/deploy_key_clone_spec.rb:98:in `block (4 levels) in <module:QA>'
# ./qa/scenario/actable.rb:14:in `perform'
# ./qa/specs/features/project/deploy_key_clone_spec.rb:94:in `block (3 levels) in <module:QA>'
# ./qa/specs/runner.rb:22:in `perform'
# ./qa/scenario/template.rb:7:in `block in perform'
# ./qa/scenario/template.rb:5:in `tap'
# ./qa/scenario/template.rb:5:in `perform'
# ./qa/scenario/test/instance.rb:22:in `perform'
# ./qa/scenario/template.rb:7:in `block in perform'
# ./qa/scenario/template.rb:5:in `tap'
# ./qa/scenario/template.rb:5:in `perform'
# ./qa/scenario/bootable.rb:14:in `launch!
The UI Job Log look like this:
But the raw log looks like:
[0KRunning with gitlab-runner 10.6.0 (a3543a27)
[0;m[0K on qa-runner-1527260152 1fb95ddd
[0;m[0KUsing Shell executor...
[0;msection_start:1527260181:prepare_script
[0Kstdin: is not a tty
Running on ed12ed0bf2e3...
section_end:1527260181:prepare_script
[0Ksection_start:1527260181:get_sources
[0Kstdin: is not a tty
[32;1mCloning repository...[0;m
Cloning into '/builds/1fb95ddd/0/gitlab-qa-sandbox/qa-test-25-05-2018-14-52-42/deploy-key-clone-project-af0560b465f5d039'...
[32;1mChecking out 89480c26 as master...[0;m
[32;1mSkipping Git submodules setup[0;m
section_end:1527260183:get_sources
[0Ksection_start:1527260183:restore_cache
[0Kstdin: is not a tty
section_end:1527260183:restore_cache
[0Ksection_start:1527260183:download_artifacts
[0Kstdin: is not a tty
section_end:1527260183:download_artifacts
[0Ksection_start:1527260183:build_script
[0Kstdin: is not a tty
[32;1m$ mkdir -p ~/.ssh[0;m
[32;1m$ ssh-keyscan -p 22 gitlab.deej.k8s-ft.win >> ~/.ssh/known_hosts[0;m
# gitlab.deej.k8s-ft.win SSH-2.0-OpenSSH_7.5
# gitlab.deej.k8s-ft.win SSH-2.0-OpenSSH_7.5
[32;1m$ eval $(ssh-agent -s)[0;m
Agent pid 79
[32;1m$ echo "$DEPLOY_KEY" | ssh-add -[0;m
Identity added: (stdin) ((stdin))
[32;1m$ git clone git@gitlab.deej.k8s-ft.win:gitlab-qa-sandbox/qa-test-25-05-2018-14-52-42/deploy-key-clone-project-af0560b465f5d039.git[0;m
Cloning into 'deploy-key-clone-project-af0560b465f5d039'...
Warning: Permanently added the ECDSA host key for IP address '35.197.110.61' to the list of known hosts.
[32;1m$ sha1sum deploy-key-clone-project-af0560b465f5d039/.gitlab-ci.yml[0;m
d60ca7464f6ec265331b51df96dcb8bb41b69dbd deploy-key-clone-project-af0560b465f5d039/.gitlab-ci.yml
section_end:1527260185:build_script
[0Ksection_start:1527260185:after_script
[0Kstdin: is not a tty
section_end:1527260185:after_script
[0Ksection_start:1527260185:archive_cache
[0Kstdin: is not a tty
section_end:1527260185:archive_cache
[0Ksection_start:1527260185:upload_artifacts
[0Kstdin: is not a tty
section_end:1527260185:upload_artifacts
[0K[32;1mJob succeeded
[0;m
Edited by DJ Mountney