gitlab-backup restore does not check for free space nor crashes/exits on disk full, it just hangs forever

Summary

During the restore process, the free disk space is not checked beforehand, if the disk space runs out, the process never crashes nor exit gracefully. It just hangs forever.

Steps to reproduce

Restore a 25GB Gitlab instance on 30GB disk

What is the current bug behavior?

Restore process hangs forever

What is the expected correct behavior?

Restore process early cancellation, crash or graceful exit on space exhaustion

Possible fixes

When restoring a Gitlab backup, the space used is more than tripled : backup archive, extracted archive, restored data. Ideally, the free space needs to be checked before unarchiving the global backup archive, and before unarchiving each sub-archive. Less ideally, the process needs to stop either gracefully or not and tell the user the disk space has run out.

Implementation Plan

We can alleviate this problem by adding a pre-flight check added when unpacking the backup.

Disk space estimation

The restore process roughly triples disk usage: the original archive + extracted contents + restored data. The check should use Sys::Filesystem (already available in the Ruby ecosystem) to compare available space against the estimated requirement:

# lib/backup/restore/unpack.rb
def check_disk_space!
  return if options.skip_storage_check? # bypass flag

  required_bytes = File.size(tar_file) * 3  # archive + extracted + restored data
  stat = Sys::Filesystem.stat(backup_path)
  available = stat.block_size * stat.blocks_available

  return if available > required_bytes

  logger.error(
    "Insufficient disk space: #{available / 1.gigabyte}GB available, " \
    "estimated #{required_bytes / 1.gigabyte}GB required. " \
    "Set BACKUP_SKIP_STORAGE_CHECK=true to bypass this check."
  )
  exit 1
end

Integration into run!

Place check_disk_space! after validate_tar_file_existence! (so tar_file is confirmed to exist before calling File.size on it) and before start_unpack!:

# lib/backup/restore/unpack.rb
def run!
  if backup_id.blank? && non_tarred_backup?
    logger.info "Non tarred backup found in #{backup_path}, using that"
    return
  end

  Dir.chdir(backup_path) do
    if backup_file_list.empty?
      no_backups_output!
    elsif many_backups?
      many_backups_output!
    end

    validate_tar_file_existence!
    check_disk_space!   # <-- added here
    start_unpack!
  end
end

Bypass flag

Pass the opt-out via an environment variable, consistent with how other backup options work. Add skip_storage_check? to Backup::Options:

# lib/backup/options.rb
def skip_storage_check?
  ENV['BACKUP_SKIP_STORAGE_CHECK'].present?
end

This keeps the check on by default while giving operators an escape hatch, which is the right default posture.

Considerations

  • Sub-archive checks: The issue description mentions checking space before each sub-archive too. This plan only covers the top-level tarball -- a follow-up could extend the check into the individual task restores.
  • Integer division truncation: available / 1.gigabyte will truncate to 0 for values under 1 GB. Maybe casting this to float or using Kernel.format("%.2f", available / 1.gigabyte.to_f) would be better?
Edited by 🤖 GitLab Bot 🤖