1. 17 Mar, 2013 1 commit
    • René Scharfe's avatar
      archive-zip: use deflateInit2() to ask for raw compressed data · c3c2e1a0
      René Scharfe authored
      We use the function git_deflate_init() -- which wraps the zlib function
      deflateInit() -- to initialize compression of ZIP file entries.  This
      results in compressed data prefixed with a two-bytes long header and
      followed by a four-bytes trailer.  ZIP file entries consist of ZIP
      headers and raw compressed data instead, so we remove the zlib wrapper
      before writing the result.
      We can ask zlib for the the raw compressed data without the unwanted
      parts in the first place by using deflateInit2() and specifying a
      negative number of bits to size the window.  For that purpose, factor
      out the function do_git_deflate_init() and add git_deflate_init_raw(),
      which wraps it.  Then use the latter in archive-zip.c and get rid of
      the code that stripped the zlib header and trailer.
      Also rename the helper function zlib_deflate() to zlib_deflate_raw()
      to reflect the change.
      Thus we avoid generating data that we throw away anyway, the code
      becomes shorter and some magic constants are removed.
      Signed-off-by: default avatarRene Scharfe <rene.scharfe@lsrfire.ath.cx>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
  2. 01 Dec, 2011 1 commit
  3. 10 Jun, 2011 6 commits
    • Junio C Hamano's avatar
      zlib: allow feeding more than 4GB in one go · e01503b5
      Junio C Hamano authored
      Update zlib_post_call() that adjusts the wrapper's notion of avail_in and
      avail_out to what came back from zlib, so that the callers can feed
      buffers larger than than 4GB to the API.
      When underlying inflate/deflate stopped processing because we fed a buffer
      larger than 4GB limit, detect that case, update the state variables, and
      let the zlib function work another round.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
    • Junio C Hamano's avatar
      zlib: zlib can only process 4GB at a time · ef49a7a0
      Junio C Hamano authored
      The size of objects we read from the repository and data we try to put
      into the repository are represented in "unsigned long", so that on larger
      architectures we can handle objects that weigh more than 4GB.
      But the interface defined in zlib.h to communicate with inflate/deflate
      limits avail_in (how many bytes of input are we calling zlib with) and
      avail_out (how many bytes of output from zlib are we ready to accept)
      fields effectively to 4GB by defining their type to be uInt.
      In many places in our code, we allocate a large buffer (e.g. mmap'ing a
      large loose object file) and tell zlib its size by assigning the size to
      avail_in field of the stream, but that will truncate the high octets of
      the real size. The worst part of this story is that we often pass around
      z_stream (the state object used by zlib) to keep track of the number of
      used bytes in input/output buffer by inspecting these two fields, which
      practically limits our callchain to the same 4GB limit.
      Wrap z_stream in another structure git_zstream that can express avail_in
      and avail_out in unsigned long. For now, just die() when the caller gives
      a size that cannot be given to a single zlib call. In later patches in the
      series, we would make git_inflate() and git_deflate() internally loop to
      give callers an illusion that our "improved" version of zlib interface can
      operate on a buffer larger than 4GB in one go.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
    • Junio C Hamano's avatar
      zlib: wrap deflateBound() too · 225a6f10
      Junio C Hamano authored
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
    • Junio C Hamano's avatar
      zlib: wrap deflate side of the API · 55bb5c91
      Junio C Hamano authored
      Wrap deflateInit, deflate, and deflateEnd for everybody, and the sole use
      of deflateInit2 in remote-curl.c to tell the library to use gzip header
      and trailer in git_deflate_init_gzip().
      There is only one caller that cares about the status from deflateEnd().
      Introduce git_deflate_end_gently() to let that sole caller retrieve the
      status and act on it (i.e. die) for now, but we would probably want to
      make inflate_end/deflate_end die when they ran out of memory and get
      rid of the _gently() kind.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
    • Junio C Hamano's avatar
      zlib: wrap inflateInit2 used to accept only for gzip format · 5e86c1fb
      Junio C Hamano authored
      http-backend.c uses inflateInit2() to tell the library that it wants to
      accept only gzip format. Wrap it in a helper function so that readers do
      not have to wonder what the magic numbers 15 and 16 are for.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
    • Junio C Hamano's avatar
      zlib wrapper: refactor error message formatter · 1a507fc1
      Junio C Hamano authored
      Before refactoring the main part of the wrappers, first move the
      logic to convert error status that come back from zlib to string
      to a helper function.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
  4. 10 Nov, 2010 1 commit