Output truncated for CI log due to outdated runner

When a job fails in a pipeline, the command output is truncated.

Example .gitlab-ci.yml:

test_issue:
  stage: test
  script:
    - echo 'can you read this?'

If a short delay is added, the output will be shown as expected:

  script:
    - echo 'can you read this?' ; sleep 5

It seems like the output buffers are periodically flushed, but are not flushed on exit.

Hey @avylove, thank you for creating this issue!

To get the right eyes on your issue more quickly, we encourage you to follow the issue triage best practices.

PLEASE NOTE: You will need to use the @gitlab-bot label command to apply labels to this issue.

To set expectations, GitLab product managers or team members can't make any promise if they will proceed with this. However, we believe everyone can contribute, and welcome you to work on this proposed change, feature or bug fix. There is a bias for action, so you don't need to wait. Try and spin up that merge request yourself.

If you need help doing so, we're always open to mentor you to drive this change.

This message was generated automatically. You're welcome to improve it.

added automation:self-triage-encouraged label

This issue was automatically tagged with the label grouppipeline execution by TanukiStan, a machine learning classification model, with a probability of 0.68.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.

If you are unsure about the correct group, please do not leave the issue without a group label. Please refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issues.

Authors who do not have permission to update labels can use the @gitlab-bot label ~"group::<correct group name>" command, or leave the issue to be triaged by group leaders initially assigned by TanukiStan.

This message was generated automatically. You're welcome to improve it.

added automation:ml grouppipeline execution labels

added devopsverify sectionci labels

mentioned in issue gitlab-org/quality/triage-reports#20001 (closed)

added typebug label

added bugperformance label

@avylove - Is this consistently reproducible? - If you can manage to record a small video/screencast it would help us debug this further

cc @jivanvl

@rutshah Yes, it's consistent. I'm not sure a video will be possible, but there is no trace anyway. Have you tried to reproduce from the example?

@avylove I created https://gitlab.com/jivanvl/log-truncation-replica/ but I couldn't replicate the issue you're mentioning

Here's a link to a pipeline with a failing job https://gitlab.com/jivanvl/log-truncation-replica/-/jobs/7958398692

cc @rutshah

@jivanvl I found the issue. Your job ran on 17.4.0~pre.110.g27400594. Mine ran on 11.2.0. I seems like the signing key wasn't installed correctly so it was pulling an old version from the Ubuntu repos. That should be removed or updated.

I also think there's some issue with the repo script for Ubuntu, but I was able to get it to work eventually and upgrade to 17.4.0. Sorry, I didn't catch that was the issue sooner. Also not sure what the issue was. I think there may have been a dependency missing for that script.

@jivanvl - Do you know if this should go to the grouprunner for the ubuntu version removal/update? or some other group.

@rutshah It seems that the issue was an outdated runner because the signing key wasn't installed correctly, but I agree with Avram; perhaps pinging grouprunner might be a good idea to let them know there are some deprecated runners across distro channels that could be worth taking down if possible

Thank you for your report @avylove, glad your issue was solved

@jivanvl @avylove a few points:

1. AFAIK Ubuntu does not include gitlab-runner in any of it's default repos. For example this returns 0 hits:

   docker run -it --rm ubuntu:latest bash -c "apt-get update && apt-cache search gitlab|grep runner"

And if they did, it would not be in our control to fix issues with their package/repo.

2. Our official channel for distributing packages is https://packages.gitlab.com/runner/gitlab-runner. Note that we only publish official releases there, and not nightly, beta or pre builds there. All this to say that I'm not sure where the 17.4.0~pre.110.g27400594 package is coming from.

@avylove can you share what repository your stale gitlab-runner package came from? Something like apt-cache madison gitlab-runner?

@avonbertoldi It's listed here https://packages.ubuntu.com/focal/gitlab-runner

@avonbertoldi The other issue is the script for setting up the official repo doesn't seem to successfully install the key. I'm not sure why, but I think it may not have had one or packages that are required.

@avylove I get a 500 response from that url, but I do see that the package is available on that release:

> docker run -it --rm ubuntu:20.04 bash -c "apt-get update && apt-cache search gitlab-runner"
...
Fetched 33.2 MB in 5s (7135 kB/s)                           
Reading package lists... Done
gitlab-runner - GitLab Runner - runs continuous integration (CI) jobs

And the version they provide is 11.2.0

> gitlab-runner --version
Version:      11.2.0
Git revision: 11.2.0
Git branch:   HEAD
GO version:   go1.10.4
Built:        unknown
OS/Arch:      linux/amd64

But as I mentioned, there's nothing we can do about that since we don't control that repo, nor did we create that package.

The other issue is the script for setting up the official repo doesn't seem to successfully install the key. I'm not sure why, but I think it may not have had one or packages that are required.

IIRC there was a bug in that script (https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh) that failed to install keys. It was fixed quite a few releases ago though, so be sure to download it again if you are running a previously downloaded version. I just ran that script in a 20.04 docker container and then did a apt-get install gitlab-runner and it did indeed install the latest (17.4.0) runner.

@avonbertoldi I'll keep an eye out for errors if I see it again. You can request Ubuntu remove the package since it's so far out of date.

removed bugperformance label

changed title from Output truncated for CI log to Output truncated for CI log due to outdated runner

changed the description

added grouprunner label and removed grouppipeline execution label

mentioned in issue gitlab-org/quality/triage-reports#20135 (closed)

I am closing this issue as the root cause is related to a runner binary sourced from a non GitLab supported project repository/

added Category:Runner Core label

closed