Skip to content

No space left on disk for `code_quality` job on Gitlab.com

Overview

There seems to be an issue when code_quality job is running on GitLab.com/SaaS recently in which a no space left on disk error is thrown due to codeclimate spinning multiple docker containers with each running its own plugin/engine and eating up disk space.

It's not immediately clear whether this is due to a specific engine taking up so much space while running or due to how codeclimate is managing those containers, but the error seem to have been reported with two different engines (i.e. rubocop and eslint) at least until now as can be seen in this thread.

Investigation

After doing a preliminarly investigation, the error seem to go away when using a runner with more disk space (i.e. saas-linux-medium-amd64) than the default hosted runner (i.e. saas-linux-small-amd64), so a recent change in hosted runners might be the root cause behind this. This can also be a workaround until the issue is resolved.

Steps to reproduce

Reproducing this could be a little tricky as it requires adding multiple projects and configurations inside the repository to ensure codeclimate is busy running its analysis engines, but since we are running into that issue ourselves in code-quality component I have used the same setup it has to reproduce the error in the following project (check failing pipeline):

https://gitlab.com/gitlab-org/secure/tests/test-code-quality-no-disk-space

Update

A project with the the following pipeline config will exhibit this error:

Click to expand 
include:
  - component: gitlab.com/components/code-quality/code-quality@main

code_quality:
  tags:
    - saas-linux-small-amd64
  before_script:
    - |
      mingb=10 # GiB <- any value smaller than 15 will fail

      # compute min space in KiB
      min=$((mingb * 1024 * 1024)) 

      # compute available space in KiB
      available=$(df -B1024 / | awk '$NF=="/" {print $4}')

      # leave only $min free KiB by allocating the difference
      fallocate -l $((available-min))k /dummy.bin

Proposal

  1. Prior to invoking codeclimate engines:install, in run.sh, compute the space available, as above
  2. If the value is less than 15GiB, log a message indicating the likelihood of failure and
    1. recommend the preferred method of calling linters directly and link to https://docs.gitlab.com/ee/ci/testing/code_quality.html#import-code-quality-results-from-a-cicd-job
  3. release image update
  4. update component: see Bump image version (components/code-quality!11 - merged) • Jason Leasure
  5. update template: see feat: Bump Code-Quality image to 0.96.0-gitlab.1 (!179580 - merged) • Jason Leasure • 17.9
Edited by Jason Leasure