HTTP/2 PUT with a File body mangles uploaded data due to use of IO.copy_stream

When trying to use HTTPX to do a presigned PUT upload to GCP cloud storage, we are encountering a problem where the bytes that get uploaded are not the same bytes that we have in the source file. This only happens if a File object is set as the request body. If we copy the data into a StringIO and pass that StringIO as the body the correct bytes get uploaded. The number of bytes uploaded is the same, but the content of the object in the bucket is not the same as source. It seems that the amount of content sent is correct, it's just that the bytes either get mangled or get sent out-of-order - so something is going wrong with HTTP/2 framing.

This happens both with Tempfiles and Files - and it has to do with the use of IO.copy_stream when encoding the body. To repro such an upload, you can do a PUT to any HTTP/2 destination with our filesize, and see what ends up on the server:

rng = Random.new(42)
tf = Tempfile.new
tf.binmode
tf.write(rng.bytes(76232))
tf.flush
tf.rewind
HTTPX.put(some_url, tf)

When we readback the uploaded data, even the first byte is incorrect:

Byte range 0..0 mismatch: "f" vs stored "\x10".
@@ -1,3 +1,3 @@
 # encoding: ASCII-8BIT
 #    valid: true
-"f"
+"\x10"

After some digging, we foudn that if we make the following change in body.rb:

# consumes and yields the request payload in chunks.
def each(&block)
  return enum_for(__method__) unless block
  return if @body.nil?

  body = stream(@body)
  if body.respond_to?(:read)
    while(chunk = body.read(1024))
      block.call(chunk)
    end
    # ::IO.copy_stream(body, ProcIO.new(block))
  elsif body.respond_to?(:each)
    body.each(&block)
  else
    block[body.to_s]
  end
end

then the test passes and the correct bytes get uploaded. I think there is something going wrong with the interaction of what happens to the output of the ProcIO (namely: data getting pulled from it using Enumerator#next and the rest) and, possibly, with HTTP/2 framing. Either way, it doesn't seem that given that the destination of copy_stream is never going to be a native IO object - that the C code of io_copy_stream can deal with in an accelerated way - the use of IO.copy_stream here is actually unwarranted. copy_stream is quite a beast of a method with a multitude of sophisticated features (like calling sendfile() under the hood), and it seems it doesn't play nice here. Moreover, because it gets a native File object as source, it actually switches into one of these "optimized code paths" as far as reading is concerned - but something doesn't work there.

Sadly I don't have time to debug further, but hope this can be helpful. I've done some exploratory testing with a local Rack-based server but with that one I get the correct bytes, so this is probably HTTP/2 framing related.

httpx v. 1.4.0 and Ruby is ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-darwin21]