kubectl logs --follow does not block in a CI job
<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "type::bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
<!-- Summarize the bug encountered concisely. -->
When `kubectl logs -f pod-name` is executed in a GitLab CI job, it returns before the watched pod exits. It returns after about 15 seconds in my environment.
### Steps to reproduce
<!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. -->
- Have a CI job configured to be able to run `kubectl` against a cluster/ns.
- Run the following Kubernetes Job there:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: test
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: test
image: bash
args:
- bash
- -c
- 'while [[ ! -f /tmp/stop ]]; do date; sleep 1; done; exit $(cat /tmp/stop)'
```
- Run the following GitLab CI job:
```yaml
Stream logs:
image:
name: bitnami/kubectl
entrypoint: [""]
stage: test
script:
- echo "streaming logs $(date)"
- kubectl --v=9 logs -f job/test
- echo "end of streaming logs $(date)"
- exit 1
```
The job should hang forever, until the pod created by the kubernetes Job terminates, but in my experience the `kubectl logs -f` command exits after about 15 seconds for some reason. This is not a problem with the cluster, running the same command locally (or anywhere else than the gitlab CI runner context, really) works as expected.
<!-- Describe what actually happens. -->
### What is the expected *correct* behavior?
`kubectl -f` should block for as long as the watched pod is alive
<!-- Describe what you should see instead. -->
### Relevant logs and/or screenshots
<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
as it's tough to read otherwise. -->
```
$ echo "streaming logs $(date)"
streaming logs Thu Sep 22 11:17:46 UTC 2022
$ kubectl logs -f job/test
Thu Sep 22 11:17:43 UTC 2022
Thu Sep 22 11:17:44 UTC 2022
Thu Sep 22 11:17:45 UTC 2022
Thu Sep 22 11:17:46 UTC 2022
Thu Sep 22 11:17:47 UTC 2022
Thu Sep 22 11:17:48 UTC 2022
Thu Sep 22 11:17:49 UTC 2022
Thu Sep 22 11:17:50 UTC 2022
Thu Sep 22 11:17:51 UTC 2022
Thu Sep 22 11:17:52 UTC 2022
Thu Sep 22 11:17:53 UTC 2022
Thu Sep 22 11:17:54 UTC 2022
Thu Sep 22 11:17:55 UTC 2022
Thu Sep 22 11:17:56 UTC 2022
Thu Sep 22 11:17:57 UTC 2022
Thu Sep 22 11:17:58 UTC 2022
Thu Sep 22 11:17:59 UTC 2022
Thu Sep 22 11:18:00 UTC 2022
Thu Sep 22 11:18:01 UTC 2022
$ echo "end of streaming logs $(date)"
end of streaming logs Thu Sep 22 11:18:02 UTC 2022
$ exit 1
```
### Output of checks
<!-- If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com -->
#### Results of GitLab environment info
<!-- Input any relevant GitLab environment information if needed. -->
<details>
<summary>Expand for output related to GitLab environment info</summary>
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.1.4
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.7
Sidekiq Version:6.4.0
Go Version: unknown
GitLab information
Version: 15.1.0
Revision: 6bea4379525
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.10
URL: https://git.rohea.com
HTTP Clone URL: https://git.rohea.com/some-group/some-project.git
SSH Clone URL: git@git.rohea.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.7.4
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
<pre>
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)
(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
</pre>
</details>
#### Results of GitLab application Check
<!-- Input any relevant GitLab application check information if needed. -->
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:check SANITIZE=true`)
(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)
(we will only investigate if the tests are passing)
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.7.4 ? ... OK (14.7.4)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
1/3 ... yes
15/4 ... yes
1/5 ... yes
1/6 ... yes
1/7 ... yes
1/8 ... yes
14/9 ... yes
27/10 ... yes
27/11 ... yes
15/12 ... yes
14/13 ... yes
15/14 ... yes
1/15 ... yes
15/16 ... yes
15/17 ... yes
15/18 ... yes
1/19 ... yes
14/20 ... yes
1/21 ... yes
1/22 ... yes
1/23 ... yes
15/24 ... yes
1/25 ... yes
15/26 ... yes
14/27 ... yes
1/28 ... yes
1/29 ... yes
15/30 ... yes
14/31 ... yes
15/32 ... yes
1/33 ... yes
27/34 ... yes
15/35 ... yes
1/36 ... yes
14/37 ... yes
27/38 ... yes
15/39 ... yes
18/40 ... yes
1/41 ... yes
15/42 ... yes
1/43 ... yes
1/44 ... yes
15/45 ... yes
15/46 ... yes
1/47 ... yes
1/48 ... yes
1/49 ... yes
14/50 ... yes
15/51 ... yes
14/52 ... yes
1/53 ... yes
1/54 ... yes
1/55 ... yes
15/56 ... yes
1/57 ... yes
1/58 ... yes
1/59 ... yes
14/60 ... yes
1/61 ... yes
15/62 ... yes
15/63 ... yes
15/64 ... yes
1/65 ... yes
14/66 ... yes
1/67 ... yes
15/68 ... yes
1/69 ... yes
15/70 ... yes
1/71 ... yes
27/72 ... yes
14/73 ... yes
1/74 ... yes
15/75 ... yes
1/76 ... yes
15/77 ... yes
27/78 ... yes
1/79 ... yes
15/80 ... yes
1/81 ... yes
14/82 ... yes
1/83 ... yes
1/84 ... yes
27/85 ... yes
14/86 ... yes
1/87 ... yes
18/88 ... yes
14/89 ... yes
15/90 ... yes
15/91 ... yes
1/92 ... yes
14/93 ... yes
15/94 ... yes
1/95 ... yes
1/96 ... yes
1/97 ... yes
1/98 ... yes
18/99 ... yes
15/100 ... yes
1/101 ... yes
1/102 ... yes
1/103 ... yes
9/105 ... yes
27/106 ... yes
15/107 ... yes
14/108 ... yes
15/109 ... yes
6/110 ... yes
15/111 ... yes
15/112 ... yes
15/113 ... yes
15/114 ... yes
15/116 ... yes
15/119 ... yes
14/120 ... yes
15/121 ... yes
9/122 ... yes
15/123 ... yes
15/124 ... yes
15/125 ... yes
15/127 ... yes
22/129 ... yes
9/130 ... yes
9/131 ... yes
15/132 ... yes
15/134 ... yes
14/135 ... yes
5/136 ... yes
14/137 ... yes
26/138 ... yes
15/150 ... yes
6/151 ... yes
15/153 ... yes
6/154 ... yes
15/155 ... yes
14/156 ... yes
15/157 ... yes
6/158 ... yes
6/159 ... yes
26/160 ... yes
14/161 ... yes
15/163 ... yes
27/165 ... yes
15/166 ... yes
14/168 ... yes
22/169 ... yes
22/170 ... yes
6/171 ... yes
15/172 ... yes
15/173 ... yes
15/174 ... yes
22/175 ... yes
14/176 ... yes
15/177 ... yes
15/178 ... yes
6/179 ... yes
6/181 ... yes
15/182 ... yes
20/183 ... yes
20/187 ... yes
20/188 ... yes
20/189 ... yes
20/190 ... yes
20/191 ... yes
20/192 ... yes
22/197 ... yes
20/198 ... yes
26/199 ... yes
15/201 ... yes
20/202 ... yes
20/204 ... yes
15/205 ... yes
20/206 ... yes
20/208 ... yes
20/209 ... yes
14/210 ... yes
14/211 ... yes
15/212 ... yes
20/213 ... yes
15/214 ... yes
26/215 ... yes
20/218 ... yes
20/219 ... yes
20/220 ... yes
15/221 ... yes
27/222 ... yes
14/224 ... yes
14/225 ... yes
27/227 ... yes
14/229 ... yes
27/230 ... yes
27/231 ... yes
15/232 ... yes
6/233 ... yes
15/234 ... yes
27/235 ... yes
27/236 ... yes
27/237 ... yes
27/238 ... yes
27/239 ... yes
27/240 ... yes
27/241 ... yes
20/242 ... yes
6/243 ... yes
22/244 ... yes
26/245 ... yes
9/246 ... yes
14/247 ... yes
15/248 ... yes
26/250 ... yes
26/251 ... yes
26/252 ... yes
14/253 ... yes
20/254 ... yes
30/255 ... yes
14/256 ... yes
14/257 ... yes
14/258 ... yes
30/259 ... yes
29/260 ... yes
20/261 ... yes
14/262 ... yes
14/263 ... yes
26/264 ... yes
14/265 ... yes
14/266 ... yes
20/267 ... yes
20/268 ... yes
6/269 ... yes
30/270 ... yes
26/271 ... yes
30/272 ... yes
20/273 ... yes
14/275 ... yes
35/276 ... yes
26/278 ... yes
26/279 ... yes
14/280 ... yes
20/281 ... yes
26/282 ... yes
14/283 ... yes
20/284 ... yes
20/285 ... yes
14/286 ... yes
30/287 ... yes
26/288 ... yes
20/291 ... yes
20/293 ... yes
14/294 ... yes
15/295 ... yes
14/296 ... yes
14/297 ... yes
20/298 ... yes
20/299 ... yes
14/300 ... yes
14/301 ... yes
14/302 ... yes
20/303 ... yes
20/304 ... yes
14/305 ... yes
20/306 ... yes
14/307 ... yes
20/308 ... yes
20/309 ... yes
14/310 ... yes
6/312 ... yes
26/313 ... yes
26/315 ... yes
44/316 ... yes
44/317 ... yes
27/318 ... yes
26/319 ... yes
44/320 ... yes
20/321 ... yes
44/322 ... yes
20/323 ... yes
44/324 ... yes
26/325 ... yes
26/326 ... yes
26/327 ... yes
26/328 ... yes
44/329 ... yes
14/330 ... yes
44/331 ... yes
14/332 ... yes
26/333 ... yes
26/334 ... yes
26/335 ... yes
44/336 ... yes
36/337 ... yes
26/338 ... yes
44/339 ... yes
50/340 ... yes
26/341 ... yes
26/342 ... yes
26/343 ... yes
55/344 ... yes
55/345 ... yes
44/346 ... yes
55/347 ... yes
57/348 ... yes
26/349 ... yes
55/350 ... yes
57/351 ... yes
57/352 ... yes
57/353 ... yes
57/354 ... yes
55/355 ... yes
55/356 ... yes
57/357 ... yes
55/358 ... yes
57/359 ... yes
55/360 ... yes
57/361 ... yes
55/362 ... yes
57/363 ... yes
44/364 ... yes
59/365 ... yes
44/366 ... yes
57/368 ... yes
44/369 ... yes
57/370 ... yes
55/371 ... yes
59/372 ... yes
57/373 ... yes
44/374 ... yes
6/375 ... yes
59/376 ... yes
26/377 ... yes
30/378 ... yes
14/379 ... yes
57/380 ... yes
55/381 ... yes
43/382 ... yes
44/383 ... yes
44/384 ... yes
44/385 ... yes
6/386 ... yes
26/387 ... yes
61/388 ... yes
61/389 ... yes
61/390 ... yes
61/391 ... yes
61/392 ... yes
61/393 ... yes
61/394 ... yes
61/395 ... yes
61/396 ... yes
55/397 ... yes
61/399 ... yes
55/400 ... yes
57/401 ... yes
61/402 ... yes
61/403 ... yes
55/404 ... yes
57/405 ... yes
14/406 ... yes
55/407 ... yes
57/408 ... yes
55/409 ... yes
57/410 ... yes
55/411 ... yes
57/412 ... yes
44/413 ... yes
18/414 ... yes
55/415 ... yes
26/416 ... yes
26/417 ... yes
26/418 ... yes
14/419 ... yes
14/420 ... yes
55/424 ... yes
57/425 ... yes
14/426 ... yes
55/427 ... yes
57/428 ... yes
14/429 ... yes
55/430 ... yes
57/431 ... yes
69/432 ... yes
44/433 ... yes
44/434 ... yes
70/435 ... yes
14/436 ... yes
55/437 ... yes
57/438 ... yes
20/439 ... yes
72/440 ... yes
26/441 ... yes
73/442 ... yes
73/443 ... yes
73/444 ... yes
73/445 ... yes
73/446 ... yes
14/448 ... yes
55/449 ... yes
57/450 ... yes
73/451 ... yes
73/452 ... yes
73/453 ... yes
26/454 ... yes
6/455 ... yes
14/456 ... yes
55/457 ... yes
57/458 ... yes
14/459 ... yes
55/460 ... yes
57/461 ... yes
81/462 ... yes
81/463 ... yes
14/464 ... yes
55/465 ... yes
57/466 ... yes
14/467 ... yes
55/468 ... yes
57/469 ... yes
73/470 ... yes
73/471 ... yes
55/474 ... yes
57/475 ... yes
26/476 ... yes
73/477 ... yes
73/478 ... yes
82/479 ... yes
26/480 ... yes
83/482 ... yes
59/483 ... yes
84/484 ... yes
83/485 ... yes
73/486 ... yes
14/487 ... yes
55/488 ... yes
57/489 ... yes
70/490 ... yes
14/491 ... yes
55/492 ... yes
57/493 ... yes
85/494 ... yes
61/495 ... yes
26/496 ... yes
73/497 ... yes
6/498 ... yes
70/499 ... yes
73/500 ... yes
73/501 ... yes
44/502 ... yes
26/503 ... yes
14/505 ... yes
20/506 ... yes
30/507 ... yes
26/508 ... yes
73/509 ... yes
73/510 ... yes
44/511 ... yes
61/512 ... yes
44/513 ... yes
59/514 ... yes
59/515 ... yes
14/516 ... yes
55/517 ... yes
57/518 ... yes
14/519 ... yes
55/520 ... yes
57/521 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 32
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
</pre>
</details>
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->
issue