<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=463696) </details> <!--IssueSummary end--> ### Summary We have a customer where `/opt/gitlab/embedded/bin/svlogd` got corrupted and was transformed into an empty file. This causes `gitlab-workhorse` to get stuck when trying to start via `gitlab-ctl start gitlab-workhorse`. Starting the workhorse process manually works. From the behavior we observed, we noticed that in the strace of the stuck process, it got stuck after trying to print the first line of the logs which includes the build date. It also appears that `gitlab-workhorse` is the only service that got broken with the corruption of `svlogd` ### Steps to reproduce 1. Stop GitLab: `gitlab-ctl stop && systemctl stop gitlab-runsvdir` 1. Truncate `/opt/gitlab/embedded/bin/svlogd`: `truncate -s 0 /opt/gitlab/embedded/bin/svlogd` 2. Restart the server. 3. `gitlab-workhorse` will not start correctly and will get stuck in this line in `strace`: ``` [pid 552789] futex(0xc000062948, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...> [pid 552788] restart_syscall(<... resuming interrupted futex ...> <unfinished ...> [pid 552786] write(2, "{\"build_time\":\"20240424.111925\","..., 118 <unfinished ...> [pid 552792] futex(0x27b8df8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...> ``` `[pid 552786] write(2, "{\"build_time\":\"20240424.111925\","..., 118 <unfinished ...>` is the first log `gitlab-workhorse` will attempt to print. ### Example Project <!-- If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version. --> ### What is the current *bug* behavior? GitLab workhorse doesn't handle failing writing to the logs correctly which causes it to get stuck ### What is the expected *correct* behavior? GitLab workhorse should process normally even if it can't write to the logs. ### Relevant logs and/or screenshots <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> <!-- This bug happens on GitLab.com --> <!-- /label ~"reproduced on GitLab.com" --> #### Results of GitLab environment info <!-- Input any relevant GitLab environment information if needed. --> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) </pre> </details> #### Results of GitLab application Check <!-- Input any relevant GitLab application check information if needed. --> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. -->