Skip to content

LFS integration tests failing on master

Since !4219 (merged) was merged, we are seeing master failures with LFS integration tests:

=== RUN   TestBuildWithGitStrategyCloneWithLFS/bash
    shell_integration_test.go:103: Build directory: /tmp/TestBuildWithGitStrategyCloneWithLFSbash3055293224/001
    test.go:24: Running with gitlab-runner 16.2.0~beta.151.g77752a6f (77752a6f)
        Preparing the "shell-integration-test" executor
        Using Shell (bash) executor...
        Preparing environment
        Running on runner-zi8oswq-project-250833-concurrent-0...
        Getting source from Git repository
        Fetching changes...
        Initialized empty Git repository in /tmp/TestBuildWithGitStrategyCloneWithLFSbash3055293224/001/0/gitlab-org/ci-cd/gitlab-runner-pipeline-tests/gitlab-test/.git/
        Created fresh repository.
        Checking out c8f2a61d as detached HEAD (ref is add-lfs-object)...
        Skipping object checkout, Git LFS is not installed for this repository.
        Consider installing it with 'git lfs install'.
        
        Skipping Git submodules setup
        Executing "step_script" stage of the job script
        $ echo Hello World
        Hello World
        Job succeeded
        
    shell_integration_test.go:442: 
        	Error Trace:	/builds/gitlab-org/gitlab-runner/executors/shell/shell_integration_test.go:442
        	            				/builds/gitlab-org/gitlab-runner/executors/shell/shell_integration_test.go:671
        	            				/builds/gitlab-org/gitlab-runner/shells/shellstest/utils.go:21
        	Error:      	Not equal: 
        	            	expected: 2097152
        	            	actual  : 132
        	Test:       	TestBuildWithGitStrategyCloneWithLFS/bash
        	Messages:   	invalid size of "/tmp/TestBuildWithGitStrategyCloneWithLFSbash3055293224/001/0/gitlab-org/ci-cd/gitlab-runner-pipeline-tests/gitlab-test/files/lfs/file_1.lfs" file

Here's an example of a working case after a retry:

The working case has this header:

Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on green-4.shared-gitlab-org.runners-manager.gitlab.com/dind 1qkDMh-h, system ID: s_b9637080a79e
  feature flags: FF_USE_FASTZIP:true, FF_USE_NEW_BASH_EVAL_STRATEGY:true, FF_SCRIPT_SECTIONS:true, FF_USE_IMPROVED_URL_MASKING:true

Failed case: https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/4701260065

Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on green-2.private.runners-manager.gitlab.com/gitlab.com/gitlab-org GaSD-S1F, system ID: s_5651e5b5643b
  feature flags: FF_NETWORK_PER_BUILD:true, FF_USE_FASTZIP:true, FF_USE_NEW_BASH_EVAL_STRATEGY:true, FF_SCRIPT_SECTIONS:true, FF_USE_IMPROVED_URL_MASKING:true

Is this a shared vs. private runner case? I'm surprised that the shared runners don't have FF_NETWORK_PER_BUILD enabled.

/cc: @ggeorgiev_gitlab @avonbertoldi

Edited by Stan Hu