## Summary Gitaly runs every Git command with `-c core.bigFileThreshold=50m`. In the Git version bundled with Gitaly 19.2.4 (`GIT_VERSION_PREV = 02bb39c5cbca65a4817854477cc3e1c31ea28c49`), `git fsck` reports a packed blob as corrupt when the blob's uncompressed size is byte-equal to `core.bigFileThreshold`. The blob is intact. No object bytes are read or hashed before the error is emitted. The failure is caused by two boundary checks in Git that disagree at `size == threshold`. In GitLab this surfaces as a persistent repository check failure (`RepositoryCheck::SingleRepositoryWorker` via the `RepositoryService.Fsck` RPC) for any repository containing a blob of exactly 52,428,800 bytes. An administrator running `git fsck` by hand on the Gitaly node gets a clean result, because the CLI uses Git's default threshold (512 MiB), which makes the false positive hard to diagnose. ## Root cause `pack-check.c`, `verify_packfile()`, decides to verify a blob through the streaming interface when: ```c if (type == OBJ_BLOB && repo_settings_get_big_file_threshold(r) <= size) { data = NULL; data_valid = 0; } else { data = unpack_entry(r, p, entries[i].offset, &type, &size); data_valid = 1; } ``` `packfile.c`, `packfile_read_object_stream()`, refuses to open a stream when: ```c if (repo_settings_get_big_file_threshold(pack->repo) >= size) return -1; ``` Both conditions are true only when `size == threshold`. `verify_packfile()` then reports the `-1` as corruption: ```c else if (!data && (packfile_read_object_stream(&stream, &oid, p, entries[i].offset) < 0 || stream_object_signature(r, stream, &oid) < 0)) err = error("packed %s from %s is corrupt", oid_to_hex(&oid), p->pack_name); ``` Nothing has inflated or hashed the object at this point. The CRC check that precedes it passes. `git verify-pack` passes. Running the same fsck with any threshold not equal to the blob size passes. Older `pack-check.c` (snapshots from 2025-01 and 2025-07) called `stream_object_signature(r, &oid)` without a pack-specific stream, and the generic stream opener fell back to in-core reading at the boundary. The ODB streaming refactor changed `verify_packfile()` to call `packfile_read_object_stream()` directly, with no fallback. This is a regression from that refactor. The disagreement is still present on upstream `master` as of 2026-09-04: - https://gitlab.com/gitlab-org/git/-/blob/master/pack-check.c (search `repo_settings_get_big_file_threshold(r) <= size`) - https://gitlab.com/gitlab-org/git/-/blob/master/packfile.c (search `repo_settings_get_big_file_threshold(pack->repo) >= size`) ## Why Gitaly is exposed Gitaly sets `core.bigFileThreshold=50m` on every Git invocation (`internal/git/gitcmd/command_factory.go`, `BigFileThresholdMB = 50`). 50 MiB is a round number that real files hit. Any repository on any Gitaly instance with a blob of exactly 52,428,800 bytes will fail `RepositoryService.Fsck` forever, across repacks, on every replica, while being healthy. Plain Git users are less exposed because the default threshold is 512 MiB and a blob of exactly 536,870,912 bytes is rarer. ## Observed on a customer instance GitLab 19.2.4, Gitaly Cluster (Praefect, 3 Gitaly nodes, Linux package), Rails and Sidekiq on the Helm chart. - Repository check failed on one project for several weeks with `error: packed <sha> from ./objects/pack/pack-<hash>.pack is corrupt`. - `git verify-pack -v` on the pack: the object is a non-delta blob with an uncompressed size of exactly 52,428,800 bytes. - The failure followed the object across two housekeeping repacks (pack filename changed, same object). - All 3 replicas at identical checksum per `gitlab:praefect:replicas`. `praefect metadata` healthy, all assigned, valid primary. - `git fsck --full --strict` and `git fsck --no-dangling` as the `git` user on all 3 Gitaly nodes: exit 0, zero errors. - Sidekiq log for the repocheck job: `gitaly_calls: 1`, `gitaly_duration_s` ~1.5, `job_status: done`. gRPC returned OK; the error text came back in `FsckResponse.error`. - Same `git fsck --no-progress --no-dangling` on all 3 nodes with Gitaly's `-c core.bigFileThreshold=50m` (and the other global `-c` flags) added: **exit 4, same error on every node**. ## Workaround Override `core.bigFileThreshold` in Gitaly's Git configuration so that no blob in the repository is byte-equal to it. Any value other than the blob size avoids the bug, because the two checks only disagree at equality. `500m` is the value already suggested in the [Gitaly troubleshooting docs](https://docs.gitlab.com/administration/gitaly/troubleshooting/) for this setting. Linux package, on every Gitaly node: ```ruby gitaly['configuration'] = { git: { config: [ { key: 'core.bigFileThreshold', value: '500m' } ] } } ``` ```sh sudo gitlab-ctl reconfigure sudo gitlab-ctl restart gitaly ``` Helm chart, in `values.yaml` under the Gitaly chart: ```yaml gitaly: git: config: - key: "core.bigFileThreshold" value: "500m" ``` Self-compiled, in `config.toml`: ```toml [[git.config]] key = "core.bigFileThreshold" value = "500m" ``` Then clear the stored repository check results and re-run the check for the affected project. Trade-off: Gitaly sets 50m so that blobs above that size are streamed rather than loaded in memory during diff and fsck. Raising it to 500m means blobs between 50 MiB and 500 MiB are read in memory again. For most repositories that is a small cost. Instances with many blobs in the hundreds of megabytes should pick a value above 50 MiB but below their largest blob, and confirm no blob is exactly equal to the chosen value. This is a configuration-level workaround. The override should be removed once the bundled Git carries the fix. ## Minimal reproduction (no Gitaly needed) ```sh git init repro && cd repro head -c 1024 /dev/urandom > blob git add blob && git commit -qm blob git repack -adq # threshold == size: false positive git -c core.bigFileThreshold=1k fsck --no-dangling; echo "exit=$?" # expected: error: packed <sha> from ./objects/pack/pack-<hash>.pack is corrupt, exit=4 # threshold < size: streams, passes git -c core.bigFileThreshold=1023 fsck --no-dangling; echo "exit=$?" # threshold > size: in-core, passes git -c core.bigFileThreshold=1025 fsck --no-dangling; echo "exit=$?" ``` ## Proposed fix One-character change in either file so the two checks agree. Either make `pack-check.c` stream only when strictly larger: ```c repo_settings_get_big_file_threshold(r) < size ``` or make `packfile.c` refuse only when strictly larger than the object: ```c repo_settings_get_big_file_threshold(pack->repo) > size ``` `index-pack.c` (`unpack_entry_data`) uses `size > threshold` for the same decision, so `pack-check.c` moving to `<` matches the existing convention elsewhere in Git. A regression test in `t1450-fsck.sh` with a blob whose size equals `core.bigFileThreshold` would prevent this coming back. ## What is needed from the Git team - Confirm and carry the fix upstream. - Backport to the bundled Git in the currently supported GitLab release lines once merged upstream. - Consider a note in the Gitaly troubleshooting docs under the existing `core.bigFileThreshold` section so Support can point administrators at it until the fix ships. ## Affected versions - Gitaly 19.2.4 confirmed (bundled Git `02bb39c5cbca65a4817854477cc3e1c31ea28c49`). - Any Gitaly release whose bundled Git includes the ODB streaming refactor of `verify_packfile()` is expected to be affected. Earliest affected release not yet pinned down.