Repository files API: `size`, `content`, and `content_sha256` describe inconsistent representations of the same file

Summary

GET /projects/:id/repository/files/:file_path returns three metadata fields that claim to describe the same file but are computed from two different byte sequences:

size — the raw Git blob byte count (from Gitaly, pre-encoding).
content — base64 of the blob bytes after Gitlab::EncodingHelper#encode! has transcoded them.
content_sha256 — SHA-256 of those same post-encoding bytes.

For any file that is not already valid UTF-8 (e.g. UTF-16, latin-1, files containing a UTF-8 BOM, etc.), encode! re-encodes the data through CharlockHolmes detection and then String#encode("UTF-16BE", undef: :replace, invalid: :replace).encode("UTF-8"), also stripping NUL bytes. The result is that:

size ≠ Base64.decode64(content).bytesize (often by a factor of ~2 for UTF-16).
content_sha256 matches the decoded content but does not match sha256sum of the file from git clone or from the UI Download button.
The /raw variants (files/:file_path/raw, blobs/:sha/raw) bypass this and return the true blob bytes — they are the only consistent way to read the canonical content via the API.

The same inconsistency exists in GET /projects/:id/repository/blobs/:sha, which exposes size and content from the same two different representations (though it does not expose a content_sha256 field).

This is reproducible today on master and is not mentioned anywhere in the repository files API docs.

Steps to reproduce

Commit a UTF-16-encoded text file (e.g. a YAML config saved from a Windows editor) to a project. Easiest way: create a file locally with iconv -f UTF-8 -t UTF-16 input.yml > config.yml and push it.

Fetch via the API:

curl --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/projects/<id>/repository/files/config%2Eyml?ref=main" \
  | jq -r '"size=\(.size) content_sha256=\(.content_sha256)"; .content' \
  | base64 -d > api_decoded.yml

Clone the repo and inspect the same file:

git clone <repo>
sha256sum repo/config.yml api_decoded.yml
wc -c repo/config.yml api_decoded.yml

Expected

sha256sum repo/config.yml == content_sha256 from the API.
wc -c api_decoded.yml == size from the API.

Actual

sha256sum repo/config.yml ≠ content_sha256 (the API hash matches the re-encoded content, not the file).
wc -c api_decoded.yml ≠ size (the API size is the raw blob size, but the decoded content is the transcoded bytes).
The UI Download button and blobs/:sha/raw return the canonical bytes that match the clone — so the inconsistency is only on the JSON endpoint.

Root cause (source-verified on `master`)

Three pieces of code make this deterministic:

lib/gitlab/git/blob.rb — size is set from Gitaly's raw entry.size in the constructor and never recomputed, while data always runs through encode!:

def initialize(options)
  %w[id name path size data mode commit_id binary].each do |key|
    self.__send__("#{key}=", options[key.to_sym])
  end
  # Retain the actual size before it is encoded
  @loaded_size = @data.bytesize if @data
  ...
  # Retain the data before it is encoded
  @raw = @data.dup
  ...
end

def data
  encode! @data
end

Note the in-code comments — the class itself already acknowledges that size and data describe different representations, and it even keeps @raw around for callers that need the original bytes.

lib/api/files.rb — both content and content_sha256 are computed from @blob.data (post-encode!), while size is @blob.size (raw):

def content_sha
  cache_client.fetch(...) do
    @blob.load_all_data!
    Digest::SHA256.hexdigest(@blob.data)   # hashes encoded bytes
  end
end

def blob_data
  { ..., size: @blob.size, encoding: "base64",
    content_sha256: content_sha, ... }
end

# show action
@blob.load_all_data!
data = blob_data
set_http_headers(data)
data.merge(content: Base64.strict_encode64(@blob.data))   # encoded

lib/gitlab/encoding_helper.rb — for a UTF-16 file the transformation chain is:

def encode!(message)
  message = force_encode_utf8(message)
  return message if message.valid_encoding?              # UTF-16 fails this
  detect = detect_encoding(message)                      # detects UTF-16
  return message.force_encoding("BINARY") if detect_binary?(message, detect)
  if detect && detect[:encoding] && detect[:confidence] > 50
    message.force_encoding(detect[:encoding])
  end
  message.replace clean(message)
end

def clean(message, replace: "")
  message.encode("UTF-16BE", undef: :replace, invalid: :replace,
                 replace: replace.encode("UTF-16BE"))
    .encode("UTF-8")
    .gsub("\0".encode("UTF-8"), "")
end

So the bytes are literally transcoded to UTF-8 and have NUL bytes stripped. Different bytes → different length → different SHA-256.

The /raw endpoints (files/:file_path/raw, blobs/:sha/raw) call send_git_blob @repo, @blob, which streams from Gitaly without ever invoking @blob.data, so they are unaffected.

Same bug class in `lib/api/repositories.rb`

GET /projects/:id/repository/blobs/:sha returns:

{ size: @blob.size, encoding: "base64", content: Base64.strict_encode64(@blob.data), sha: @blob.id }

size is raw, content is encoded. No content_sha256 is exposed, but size still does not describe the decoded content bytes.

Proposal

Two reasonable directions, not mutually exclusive:

Make the JSON response self-consistent (preferred). Gitlab::Git::Blob already keeps @raw precisely for this kind of caller. The files endpoint should base64 @blob.raw (or @raw exposed via a public accessor) and hash that same byte string for content_sha256. size already matches the raw bytes, so all three fields would describe the same representation, matching git clone and the UI Download button. The same change should be applied to lib/api/repositories.rb's blobs/:sha endpoint.

Risk: existing clients that decode content and expect already-UTF-8 text would now have to handle the original encoding. This is arguably the correct behavior (the field is documented as base64-encoded binary), but it is a behavior change and would warrant a deprecation note.
At minimum, document the current behavior. Update doc/api/repository_files.md to clarify:
- content is transcoded to UTF-8 via Gitlab::EncodingHelper#encode! and is not guaranteed to byte-match the underlying Git blob.
- content_sha256 is the SHA-256 of the (transcoded) content, not of the raw blob.
- size is the raw Git blob byte count and may not equal Base64.decode64(content).bytesize.
- To obtain the canonical blob bytes, use GET /projects/:id/repository/files/:file_path/raw or GET /projects/:id/repository/blobs/:sha/raw.

Option 2 alone unblocks users who currently have no way to know the field is transcoded; option 1 is what most users actually want.

Notes

The relevant fields have looked this way for a long time; this is a long-standing inconsistency rather than a recent regression.

/cc the source-code-management group.

Repository files API: size, content, and content_sha256 describe inconsistent representations of the same file