Ocamlnet 4.1.6 - Buffer closed before really being flushed when using netzip

Thierry Caebergs
2021-03-23T19:35:14Z

Dear developers,

I have been affected by a bug when using zip compression when retrieving files with ocamlnet.
Some were cut, some other not.
After investigation, I realized that it is linked to the file size, and more specifically, problems dealing with the separation in data chunk from the connection (--> equeue) :

If one chunk of zip content was needed : OK

If two chunks of zip content were needed : it was truncated to the content of the first chunk (decompressed : 1024B)

If more chunks of zip content were needed : it was OK (for decompressed files about bigger than 35kB)
It seems to me that the buffer was closed before being fully read in some circumstances.

I narrowed down the problem to equeue/uq_io.ml.
Please find the diff file here below. It worked for me on release 4.1.6.
Only changes in equeue/uq_io.ml seem to be necessary.

Could you please review this patch and apply it for the next release if suitable ?

Thank you in advance !

Best regards,

Thierry

chunked_gzip_compression.patch

PS : Here are the main lines concerning the ocamlnet setup in my code.
In the main() :

  Netgzip.init();
  Nettls_gnutls.init();

For each call :

let call = new H.post "https://xyz.com" [("aaa" , "xxx") ; ("bbb" , "yyy")] in
  call # set_accept_encoding () ;
  call # set_response_body_storage (`File (fun () ->"test.dat")) ;
  call # set_chunked_request () ;`
  pipeline # add call;
  pipeline # run();