All jobs failing in e2e-package-and-test due to error during `docker attach`

Occurred in pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/1185986480 Example job: https://gitlab.com/gitlab-org/gitlab/-/jobs/6230096044

Stack trace:

[Feb 22 2024 05:20:04 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitlab-ee-7b41431a`
[Feb 22 2024 05:20:09 UTC (Gitlab QA)] WARN  -- Retry instance_no_teardown due to Support::ShellCommand::StatusError -- attempt 1
[Feb 22 2024 05:20:09 UTC (Gitlab QA)] DEBUG -- Fetching pg statistics
[Feb 22 2024 05:20:09 UTC (Gitlab QA)] INFO  -- Shell command: `docker exec gitlab-ee-7b41431a bash -c "gitlab-psql -c 'select n_live_tup, n_dead_tup, relname from pg_stat_all_tables order by n_live_tup DESC, n_dead_tup DESC;'"`
[Feb 22 2024 05:20:09 UTC (Gitlab QA)] ERROR -- Shell command output:
Error response from daemon: Container 35c815fc4b213723b28ab01355e96dcb7a002d0bab8a0cc384e4d4207baa5952 is not running
[Feb 22 2024 05:20:09 UTC (Gitlab QA)] ERROR -- Error getting pg statistics: Command `docker exec gitlab-ee-7b41431a bash -c "gitlab-psql -c 'select n_live_tup, n_dead_tup, relname fr...` failed! ✘

The error seems to occur when doing docker attach, the error is caught and then the command is retried 3 times. The last error occurs because the container isn't running and is expected.

To compare, here are the logs for a successful job:

[Feb 22 2024 03:28:51 UTC (Gitlab QA)] INFO  -- Shell command: `docker attach --sig-proxy=false gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:22 UTC (Gitlab QA)] INFO  -- Waiting for GitLab to become healthy ...
[Feb 22 2024 03:32:22 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:23 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:24 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:25 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:26 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:27 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:28 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:29 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:30 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:31 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:32 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:33 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:34 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:35 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:36 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:37 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:40 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:41 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:42 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:43 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:44 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:45 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:46 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:47 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:48 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:49 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:50 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:51 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:52 UTC (Gitlab QA)] INFO  -- Shell command: `docker inspect --format='{{json .State.Health.Status}}' gitlab-ee-cf56ab92`
[Feb 22 2024 03:32:52 UTC (Gitlab QA)] INFO  -- Shell command: `docker port gitlab-ee-cf56ab92 80/tcp`
[Feb 22 2024 03:32:52 UTC (Gitlab QA)] INFO  -- -> GitLab is available at `http://docker:32768`!

Edited Feb 22, 2024 by Jay McCure