gitlab-backup restore does not check for free space nor crashes/exits on disk full, it just hangs forever
Summary
During the restore process, the free disk space is not checked beforehand, if the disk space runs out, the process never crashes nor exit gracefully. It just hangs forever.
Steps to reproduce
Restore a 25GB Gitlab instance on 30GB disk
What is the current bug behavior?
Restore process hangs forever
What is the expected correct behavior?
Restore process early cancellation, crash or graceful exit on space exhaustion
Possible fixes
When restoring a Gitlab backup, the space used is more than tripled : backup archive, extracted archive, restored data. Ideally, the free space needs to be checked before unarchiving the global backup archive, and before unarchiving each sub-archive. Less ideally, the process needs to stop either gracefully or not and tell the user the disk space has run out.
Implementation Plan
We can alleviate this problem by adding a pre-flight check added when unpacking the backup.
Disk space estimation
The restore process roughly triples disk usage: the original archive + extracted contents + restored data. The check should use Sys::Filesystem (already available in the Ruby ecosystem) to compare available space against the estimated requirement:
# lib/backup/restore/unpack.rb
def check_disk_space!
return if options.skip_storage_check? # bypass flag
required_bytes = File.size(tar_file) * 3 # archive + extracted + restored data
stat = Sys::Filesystem.stat(backup_path)
available = stat.block_size * stat.blocks_available
return if available > required_bytes
logger.error(
"Insufficient disk space: #{available / 1.gigabyte}GB available, " \
"estimated #{required_bytes / 1.gigabyte}GB required. " \
"Set BACKUP_SKIP_STORAGE_CHECK=true to bypass this check."
)
exit 1
endIntegration into run!
Place check_disk_space! after validate_tar_file_existence! (so tar_file is confirmed to exist before calling File.size on it) and before start_unpack!:
# lib/backup/restore/unpack.rb
def run!
if backup_id.blank? && non_tarred_backup?
logger.info "Non tarred backup found in #{backup_path}, using that"
return
end
Dir.chdir(backup_path) do
if backup_file_list.empty?
no_backups_output!
elsif many_backups?
many_backups_output!
end
validate_tar_file_existence!
check_disk_space! # <-- added here
start_unpack!
end
endBypass flag
Pass the opt-out via an environment variable, consistent with how other backup options work. Add skip_storage_check? to Backup::Options:
# lib/backup/options.rb
def skip_storage_check?
ENV['BACKUP_SKIP_STORAGE_CHECK'].present?
endThis keeps the check on by default while giving operators an escape hatch, which is the right default posture.
Considerations
- Sub-archive checks: The issue description mentions checking space before each sub-archive too. This plan only covers the top-level tarball -- a follow-up could extend the check into the individual task restores.
- Integer division truncation:
available / 1.gigabytewill truncate to 0 for values under 1 GB. Maybe casting this to float or usingKernel.format("%.2f", available / 1.gigabyte.to_f)would be better?