## Summary Backup restore fails with `gtar: .: Cannot unlink: Invalid argument` on systems running RHEL/EL-patched GNU tar (`tar-1.34-9.el9_7` or later). Reported on the forums: https://forum.gitlab.com/t/cant-restore-18-7-1/132444 ## Root Cause [RHSA-2026:0067](https://access.redhat.com/errata/RHSA-2026:0067) patches GNU tar for [CVE-2025-45582](https://nvd.nist.gov/vuln/detail/CVE-2025-45582) (a two-archive symlink path traversal). The CVE fix itself uses `openat2(RESOLVE_BENEATH)` to jailify the extraction directory and does **not** restrict `--unlink-first`. However, the same advisory bundles a separate fix ([RHEL-136277](https://issues.redhat.com/browse/RHEL-136277)) that changes how tar handles `unlink(".")`. On XFS (and possibly other filesystems), `unlink(".")` now returns `EINVAL`, and the patched tar propagates this as a fatal error instead of ignoring it. GitLab backup archives contain `./` as the root entry. When `--unlink-first` is passed, tar attempts `unlink(".")` on that entry, triggering the fatal error on patched systems. - CVE: https://nvd.nist.gov/vuln/detail/CVE-2025-45582 - Red Hat advisory: https://access.redhat.com/errata/RHSA-2026:0067 - GNU tar maintainer response: https://lists.gnu.org/archive/html/bug-tar/2025-08/msg00012.html Affected file: `gems/gitlab-backup-cli/lib/gitlab/backup/cli/utils/tar.rb` (lines 63-72) ## Affected Versions - GitLab 18.5.5+ - Confirmed on EL9 and EL10 (Rocky Linux, RHEL, AlmaLinux) with `tar-1.34-9.el9_7` or later - Not affected: Ubuntu, Debian, or other distros whose tar packages do not include the RHEL-136277 fix ## Workaround Downgrade tar package to version 1.34-7.el9 or earlier. ## Error Log ``` 2026-01-23 09:30:10 UTC -- Deleting backup and restore PID file at [/opt/gitlab/embedded/service/gitlab-rails/tmp/backup_restore.pid] ... done rake aborted! Backup::Error: Restore operation failed: gtar: .: Cannot unlink: Invalid argument gtar: Exiting with failure status due to previous errors /opt/gitlab/embedded/service/gitlab-rails/lib/backup/targets/files.rb:104:in `restore' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/tasks/task.rb:31:in `restore!' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/restore/process.rb:30:in `execute!' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:101:in `run_restore_task' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:168:in `block in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:165:in `each_value' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:165:in `run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:68:in `restore' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rb:22:in `block in restore_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rb:83:in `lock_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rb:19:in `restore_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:14:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => gitlab:backup:restore (See full trace by running task with --trace ``` ## Proposed solution The `backup_existing_files_dir` method in `lib/backup/targets/files.rb:107-126` already moves all existing files to a timestamped backup directory **before** tar extraction runs. The `--unlink-first` and `--recursive-unlink` flags are redundant and should be removed. The flags were added in https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17516. > **Note — Backup CLI gem behavior change:** The shared `Tar#extract_cmd` method is also used by the backup CLI gem's restore path (`gems/gitlab-backup-cli/lib/gitlab/backup/cli/targets/files.rb`). Unlike the Rails restore path, the gem's restore does **not** pre-move existing files (tracked in https://gitlab.com/gitlab-org/gitlab/-/issues/499876 — it currently logs a warning and continues). Removing the flags means the gem path will no longer force-unlink existing files before extraction. This is acceptable because: > 1. The gem path already lacks proper pre-move logic, so the flags only provided partial coverage. > 2. Tar's default behavior (overwrite existing files) is sufficient for the gem's current "warn and continue" approach. > 3. When https://gitlab.com/gitlab-org/gitlab/-/issues/499876 is resolved, the gem will have proper pre-move logic matching the Rails path. ## Implementation Plan 1. **Edit** `gems/gitlab-backup-cli/lib/gitlab/backup/cli/utils/tar.rb` - Remove `--unlink-first` and `--recursive-unlink` from `extract_cmd` (lines 64-66) 2. **Update spec** `spec/lib/backup/targets/files_spec.rb` - Remove assertions expecting the deleted flags (lines 76-87) 3. **Run tests** ```shell bundle exec rspec spec/lib/backup/targets/files_spec.rb bundle exec rspec gems/gitlab-backup-cli/spec/ ### Plan to Validate the Proposed Solution <details> <summary>Click to expand plan generated with Duo (corrected after execution)</summary> Here is a compact end-to-end runbook you can execute locally. > **Important notes from execution:** > - The GitLab container is Ubuntu 24.04 (not RHEL). Ubuntu's tar 1.35 is **unpatched** and does not exhibit the bug. Step 6 uses a wrapper script to simulate the RHEL-patched tar behavior instead of `apt-get install`. > - Files created by root in the uploads directory will cause `Errno::EACCES` during restore because `backup_existing_files_dir` runs as the `git` user. Step 3 sets ownership to `git:git`. #### End-to-end Docker validation flow 1. **Start GitLab container** ```bash docker run --detach \ --hostname gitlab.local \ --publish 8929:8929 --publish 2222:22 \ --name gitlab-restore-test \ --restart unless-stopped \ --volume gitlab_config:/etc/gitlab \ --volume gitlab_logs:/var/log/gitlab \ --volume gitlab_data:/var/opt/gitlab \ gitlab/gitlab-ee:18.8.3-ee.0 ``` 2. **Wait for readiness** ```bash docker exec -it gitlab-restore-test gitlab-ctl status docker exec -it gitlab-restore-test gitlab-rake gitlab:check SANITIZE=true ``` 3. **Create sample data (minimum)** ```bash docker exec -it gitlab-restore-test bash -lc ' set -e mkdir -p /var/opt/gitlab/gitlab-rails/uploads/validation-seed echo "seed" > /var/opt/gitlab/gitlab-rails/uploads/validation-seed/file.txt chown -R git:git /var/opt/gitlab/gitlab-rails/uploads/validation-seed ' ``` > **Why `chown`?** The restore process calls `backup_existing_files_dir` which runs `FileUtils.mv` as the `git` user. If files in the uploads directory are owned by root, the move fails with `Errno::EACCES` before tar extraction even runs. This is unrelated to the tar flag issue being tested. 4. **Create backup** ```bash docker exec -it gitlab-restore-test gitlab-backup create STRATEGY=copy docker exec -it gitlab-restore-test bash -lc 'ls -1 /var/opt/gitlab/backups/*_gitlab_backup.tar' ``` 5. **Run baseline restore script (should pass before tar change)** ```bash docker exec -it gitlab-restore-test bash -lc ' cat >/root/validate_restore_min.sh << "EOF" #!/usr/bin/env bash set -euo pipefail BID="$(ls -1 /var/opt/gitlab/backups/*_gitlab_backup.tar | sed "s#.*/##" | sed "s/_gitlab_backup.tar$//" | sort | tail -1)" UP="/var/opt/gitlab/gitlab-rails/uploads" mkdir -p "$UP/preexisting" echo dummy > "$UP/preexisting/should_move.txt" chown -R git:git "$UP/preexisting" echo "=== Restore 1 ===" gitlab-backup restore BACKUP="$BID" force=yes 2>&1 | tee /root/restore1.log if [ -f "$UP/preexisting/should_move.txt" ]; then echo "FAIL: preexisting file was NOT moved" exit 1 fi echo "=== Restore 2 (idempotency) ===" gitlab-backup restore BACKUP="$BID" force=yes 2>&1 | tee /root/restore2.log if grep -qE "Cannot unlink|Backup::Error|Exiting with failure status" /root/restore1.log /root/restore2.log; then echo "FAIL: Found error patterns in restore logs" exit 1 fi echo "PASS" EOF chmod +x /root/validate_restore_min.sh /root/validate_restore_min.sh ' ``` 6. **Simulate RHEL-patched tar behavior (the breaking condition)** The GitLab container runs Ubuntu 24.04 with unpatched GNU tar 1.35. The breakage is caused by a RHEL-specific patch (RHEL-136277) bundled in RHSA-2026:0067 that makes `unlink(".")` fatal. Since we cannot install the RHEL-patched tar on Ubuntu, we use a wrapper script that simulates the patched behavior: extract operations that combine `--extract` with `--unlink-first` fail with the same error message. ```bash docker exec -it gitlab-restore-test bash -lc ' set -e # Back up real tar cp /usr/bin/tar /usr/bin/tar.real # Create wrapper that simulates RHEL-patched tar behavior cat > /usr/bin/tar << "WRAPPER" #!/usr/bin/env bash REAL_TAR=/usr/bin/tar.real HAS_EXTRACT=false HAS_UNLINK_FIRST=false for arg in "$@"; do case "$arg" in --extract|-x) HAS_EXTRACT=true ;; --unlink-first|-U) HAS_UNLINK_FIRST=true ;; esac done # Simulate RHEL-patched behavior: fail when extract + unlink-first if $HAS_EXTRACT && $HAS_UNLINK_FIRST; then echo "tar: .: Cannot unlink: Invalid argument" >&2 echo "tar: Exiting with failure status due to previous errors" >&2 exit 2 fi exec $REAL_TAR "$@" WRAPPER chmod +x /usr/bin/tar tar --version | head -1 echo "RHEL-patched tar simulator installed" ' ``` 7. **Show breakage on current code** ```bash docker exec -it gitlab-restore-test bash -lc '/root/validate_restore_min.sh || true' docker exec -it gitlab-restore-test bash -lc 'grep -E "Cannot unlink|Backup::Error" -n /root/restore1.log /root/restore2.log || true' ``` 8. **Hotpatch `tar.rb` (remove the two flags)** ```bash docker exec -it gitlab-restore-test bash -lc ' set -e FILE=/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-backup-cli/lib/gitlab/backup/cli/utils/tar.rb cp "$FILE" "${FILE}.bak" sed -i "/--unlink-first/d; /--recursive-unlink/d" "$FILE" grep -n "unlink-first\|recursive-unlink" "$FILE" || echo "Flags removed successfully" ' ``` 9. **Re-run minimum validation after patch (should pass)** ```bash docker exec -it gitlab-restore-test bash -lc '/root/validate_restore_min.sh' ``` 10. **Nested-path smoke test** ```bash docker exec -it gitlab-restore-test bash -lc ' set -e echo "gitlab_rails[\"uploads_directory\"] = \"/var/opt/gitlab/data/shared/gitlab/uploads\"" >> /etc/gitlab/gitlab.rb gitlab-ctl reconfigure mkdir -p /var/opt/gitlab/data/shared/gitlab/uploads/nested-seed echo nested > /var/opt/gitlab/data/shared/gitlab/uploads/nested-seed/file.txt chown -R git:git /var/opt/gitlab/data/shared/gitlab/uploads gitlab-backup create STRATEGY=copy /root/validate_restore_min.sh ' ``` 11. **Permission failure test (fail-fast behavior)** ```bash docker exec -it gitlab-restore-test bash -lc ' set -euo pipefail BID="$(ls -1 /var/opt/gitlab/backups/*_gitlab_backup.tar | sed "s#.*/##" | sed "s/_gitlab_backup.tar$//" | sort | tail -1)" UP="/var/opt/gitlab/gitlab-rails/uploads" mkdir -p "$UP/perm-test" echo blocked > "$UP/perm-test/file.txt" # Make directory effectively non-movable for restore user path handling chmod 000 "$UP/perm-test" set +e gitlab-backup restore BACKUP="$BID" force=yes | tee /root/restore-perm.log RC=$? set -e # Cleanup perms so container is usable chmod 755 "$UP/perm-test" || true if [ "$RC" -eq 0 ]; then echo "FAIL: expected restore failure due to permission block" exit 1 fi grep -Ei "permission denied|cannot|failed|error|Backup::Error" /root/restore-perm.log >/dev/null echo "PASS: restore failed fast with actionable error" ' ``` 12. **Data integrity spot-check (sample checksums)** ```bash docker exec -it gitlab-restore-test bash -lc ' set -euo pipefail UP="/var/opt/gitlab/gitlab-rails/uploads" BID="$(ls -1 /var/opt/gitlab/backups/*_gitlab_backup.tar | sed "s#.*/##" | sed "s/_gitlab_backup.tar$//" | sort | tail -1)" mkdir -p "$UP/integrity" chown git:git "$UP/integrity" for i in 1 2 3 4 5; do echo "sample-$i-$(date +%s)" > "$UP/integrity/file$i.txt" chown git:git "$UP/integrity/file$i.txt" done find "$UP/integrity" -type f | sort | xargs sha256sum > /root/integrity-before.sha256 gitlab-backup create STRATEGY=copy >/root/integrity-backup.log BID2="$(ls -1 /var/opt/gitlab/backups/*_gitlab_backup.tar | sed "s#.*/##" | sed "s/_gitlab_backup.tar$//" | sort | tail -1)" # Mutate files so restore must actually recover old content for i in 1 2 3 4 5; do echo "mutated-$i" > "$UP/integrity/file$i.txt"; done chown -R git:git "$UP/integrity" gitlab-backup restore BACKUP="$BID2" force=yes | tee /root/integrity-restore.log find "$UP/integrity" -type f | sort | xargs sha256sum > /root/integrity-after.sha256 diff -u /root/integrity-before.sha256 /root/integrity-after.sha256 echo "PASS: integrity sample matches after restore" ' ``` 13. **Collect evidence** ```bash docker exec -it gitlab-restore-test bash -lc ' tar --version ls -l /root/restore*.log ' ``` This gives you: breakage reproduced, hotpatch verified, and minimum regression confidence (preexisting-data handling + repeat restore; plus optional nested-path check). </details>