1. 21 Sep, 2018 1 commit
  2. 13 Aug, 2018 1 commit
  3. 23 Jul, 2018 2 commits
  4. 16 May, 2018 1 commit
    • Stefan Beller's avatar
      object-store: move object access functions to object-store.h · cbd53a21
      Stefan Beller authored
      This should make these functions easier to find and cache.h less
      overwhelming to read.
      
      In particular, this moves:
      - read_object_file
      - oid_object_info
      - write_object_file
      
      As a result, most of the codebase needs to #include object-store.h.
      In this patch the #include is only added to files that would fail to
      compile otherwise.  It would be better to #include wherever
      identifiers from the header are used.  That can happen later
      when we have better tooling for it.
      Signed-off-by: Stefan Beller's avatarStefan Beller <sbeller@google.com>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      cbd53a21
  5. 26 Apr, 2018 1 commit
  6. 14 Mar, 2018 4 commits
  7. 15 Jun, 2017 1 commit
  8. 28 Apr, 2017 1 commit
  9. 27 Apr, 2017 1 commit
    • Johannes Schindelin's avatar
      timestamp_t: a new data type for timestamps · dddbad72
      Johannes Schindelin authored
      Git's source code assumes that unsigned long is at least as precise as
      time_t. Which is incorrect, and causes a lot of problems, in particular
      where unsigned long is only 32-bit (notably on Windows, even in 64-bit
      versions).
      
      So let's just use a more appropriate data type instead. In preparation
      for this, we introduce the new `timestamp_t` data type.
      
      By necessity, this is a very, very large patch, as it has to replace all
      timestamps' data type in one go.
      
      As we will use a data type that is not necessarily identical to `time_t`,
      we need to be very careful to use `time_t` whenever we interact with the
      system functions, and `timestamp_t` everywhere else.
      Signed-off-by: Johannes Schindelin's avatarJohannes Schindelin <johannes.schindelin@gmx.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      dddbad72
  10. 25 Apr, 2017 4 commits
    • René Scharfe's avatar
      archive-zip: support files bigger than 4GB · 4cdf3f9d
      René Scharfe authored
      Write a zip64 extended information extra field for big files as part of
      their local headers and as part of their central directory headers.
      Also write a zip64 version of the data descriptor in that case.
      
      If we're streaming then we don't know the compressed size at the time we
      write the header.  Deflate can end up making a file bigger instead of
      smaller if we're unlucky.  Write a local zip64 header already for files
      with a size of 2GB or more in this case to be on the safe side.
      
      Both sizes need to be included in the local zip64 header, but the extra
      field for the directory must only contain 64-bit equivalents for 32-bit
      values of 0xffffffff.
      Signed-off-by: default avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      4cdf3f9d
    • René Scharfe's avatar
      archive-zip: support archives bigger than 4GB · af95749f
      René Scharfe authored
      Add a zip64 extended information extra field to the central directory
      and emit the zip64 end of central directory records as well as locator
      if the offset of an entry within the archive exceeds 4GB.
      Signed-off-by: default avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      af95749f
    • René Scharfe's avatar
      archive-zip: write ZIP dir entry directly to strbuf · 3c78fd80
      René Scharfe authored
      Write all fields of the ZIP directory record for an archive entry
      in the right order directly into the strbuf instead of taking a detour
      through a struct.  Do that at end, when we have all necessary data like
      checksum and compressed size.  The fields are documented just as well,
      the code becomes shorter and we save an extra copy.
      Signed-off-by: default avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      3c78fd80
    • René Scharfe's avatar
      archive-zip: use strbuf for ZIP directory · c061a149
      René Scharfe authored
      Keep the ZIP central directory, which is written after all archive
      entries, in a strbuf instead of a custom-managed buffer.  It contains
      binary data, so we can't (and don't want to) use the full range of
      strbuf functions and we don't need the terminating NUL, but the result
      is shorter and simpler code.
      Signed-off-by: default avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      c061a149
  11. 08 Jan, 2017 1 commit
    • Jeff King's avatar
      archive-zip: load userdiff config · 965cba2e
      Jeff King authored
      Since 4aff646d (archive-zip: mark text files in archives,
      2015-03-05), the zip archiver will look at the userdiff
      driver to decide whether a file is text or binary. This
      usually doesn't need to look any further than the attributes
      themselves (e.g., "-diff", etc). But if the user defines a
      custom driver like "diff=foo", we need to look at
      "diff.foo.binary" in the config. Prior to this patch, we
      didn't actually load it.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Acked-by: default avatarRené Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      965cba2e
  12. 28 Aug, 2015 2 commits
  13. 14 Mar, 2015 1 commit
  14. 05 Mar, 2015 2 commits
    • René Scharfe's avatar
      zlib: initialize git_zstream in git_deflate_init{,_gzip,_raw} · 9a6f1287
      René Scharfe authored
      Clear the git_zstream variable at the start of git_deflate_init() etc.
      so that callers don't have to do that.
      Signed-off-by: default avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      9a6f1287
    • René Scharfe's avatar
      archive-zip: mark text files in archives · 4aff646d
      René Scharfe authored
      Set the text flag for ZIP archive entries that look like text files so
      that unzip -a can be used to perform end-of-line conversions.  Info-ZIP
      zip does the same.
      
      Detect binary files the same way as git diff and git grep do, namely by
      checking for the attribute "diff" and its negation "-diff", and if none
      is found by falling back to checking for the presence of NUL bytes in
      the first few bytes of the file contents.
      
      7-Zip, Windows' built-in ZIP functionality and Info-ZIP unzip without
      the switch -a are not affected by the change and still extract text
      files without doing any end-of-line conversions.
      
      NB: The actual end-of-line style used in the archive entries doesn't
      matter to unzip -a, as it converts any CR, CRLF and LF to the line end
      characters appropriate for the platform it is running on.
      Suggested-by: default avatarUlrike Fischer <luatex@nililand.de>
      Signed-off-by: default avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      4aff646d
  15. 29 May, 2013 1 commit
  16. 17 Mar, 2013 1 commit
    • René Scharfe's avatar
      archive-zip: use deflateInit2() to ask for raw compressed data · c3c2e1a0
      René Scharfe authored
      We use the function git_deflate_init() -- which wraps the zlib function
      deflateInit() -- to initialize compression of ZIP file entries.  This
      results in compressed data prefixed with a two-bytes long header and
      followed by a four-bytes trailer.  ZIP file entries consist of ZIP
      headers and raw compressed data instead, so we remove the zlib wrapper
      before writing the result.
      
      We can ask zlib for the the raw compressed data without the unwanted
      parts in the first place by using deflateInit2() and specifying a
      negative number of bits to size the window.  For that purpose, factor
      out the function do_git_deflate_init() and add git_deflate_init_raw(),
      which wraps it.  Then use the latter in archive-zip.c and get rid of
      the code that stripped the zlib header and trailer.
      
      Also rename the helper function zlib_deflate() to zlib_deflate_raw()
      to reflect the change.
      
      Thus we avoid generating data that we throw away anyway, the code
      becomes shorter and some magic constants are removed.
      Signed-off-by: default avatarRene Scharfe <rene.scharfe@lsrfire.ath.cx>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      c3c2e1a0
  17. 27 Feb, 2013 1 commit
    • René Scharfe's avatar
      archive-zip: fix compressed size for stored export-subst files · d3c1472f
      René Scharfe authored
      Currently ZIP archive entries of files with export-subst attribute are
      broken if they are stored uncompressed.
      
      We get the size of a file from sha1_object_info(), but this number is
      likely wrong for files whose contents are changed due to export-subst
      placeholder expansion.  We use sha1_file_to_archive() to get the
      expanded file contents and size in that case.  We proceed to use that
      size for the uncompressed size field (good), but the compressed size
      field is set based on the size from sha1_object_info() (bad).
      
      This matters only for uncompressed files because for deflated files
      we use the correct value after compression is done.  And for files
      without export-subst expansion the sizes from sha1_object_info() and
      sha1_file_to_archive() are the same, so they are unaffected as well.
      
      This patch fixes the issue by setting the compressed size based on the
      uncompressed size only after we actually know the latter.
      
      Also make use of the test file substfile1 to check for the breakage;
      it was only stored verbatim so far.  For that purpose, set the
      attribute export-subst and replace its contents with the expected
      expansion after committing.
      Signed-off-by: default avatarRene Scharfe <rene.scharfe@lsrfire.ath.cx>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      d3c1472f
  18. 06 Jan, 2013 1 commit
    • René Scharfe's avatar
      archive-zip: write uncompressed size into header even with streaming · 5ea2c847
      René Scharfe authored
      We record the uncompressed and compressed sizes and the CRC of streamed
      files as zero in the local header of the file.  The actual values are
      recorded in an extra data descriptor after the file content, and in the
      usual ZIP directory entry at the end of the archive.
      
      While we know the compressed size and the CRC only after we processed
      the contents, we actually know the uncompressed size right from the
      start.  And for files that we store uncompressed we also already know
      their final size.
      
      Do it like InfoZIP's zip and recored the known values, even though they
      can be reconstructed using the ZIP directory and the data descriptors
      alone.  InfoZIP's unzip worked fine before, but NetBSD's version
      actually depends on these fields.
      
      The uncompressed size is already set by sha1_object_info().  We just
      need to initialize the compressed size to zero or the uncompressed size
      depending on the compression method (0 means storing).  The CRC was
      propertly initialized already.
      Signed-off-by: default avatarRene Scharfe <rene.scharfe@lsrfire.ath.cx>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      5ea2c847
  19. 24 Sep, 2012 1 commit
    • René Scharfe's avatar
      archive-zip: write extended timestamp · 227bf598
      René Scharfe authored
      File modification times in ZIP files are encoded in DOS format: local
      time with a granularity of two seconds.  Add an extra field to all
      archive entries to also record the mtime in Unix' fashion, as UTC with
      a granularity of one second.
      
      This has the desirable side-effect of convincing Info-ZIP unzip 6.00
      to respect general purpose flag 11, which is used to indicate that a
      file name is encoded in UTF-8.  Any extra field would do, actually,
      but the extended timestamp is a reasonably small one (22 bytes per
      entry).  Archives created by Info-ZIP zip 3.0 contain it, too (but
      with ctime and atime as well).
      Signed-off-by: default avatarRene Scharfe <rene.scharfe@lsrfire.ath.cx>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      227bf598
  20. 18 Sep, 2012 2 commits
  21. 04 Sep, 2012 1 commit
    • René Scharfe's avatar
      archive-zip: support UTF-8 paths · 2162bd8c
      René Scharfe authored
      Set general purpose flag 11 if we encounter a path that contains
      non-ASCII characters.  We assume that all paths are given as UTF-8; no
      conversion is done.
      
      The flag seems to be ignored by unzip unless we also mark the archive
      entry as coming from a Unix system.  This is done by setting the field
      creator_version ("version made by" in the standard[1]) to 0x03NN.
      
      The NN part represents the version of the standard supported by us, and
      this patch sets it to 3f (for version 6.3) for Unix paths.  We keep
      creator_version set to 0 (FAT filesystem, standard version 0) in the
      non-special cases, as before.
      
      But when we declare a file to have a Unix path, then we have to set the
      file mode as well, or unzip will extract the files with the permission
      set 0000, i.e. inaccessible by all.
      
      [1] http://www.pkware.com/documents/casestudies/APPNOTE.TXTSigned-off-by: default avatarRene Scharfe <rene.scharfe@lsrfire.ath.cx>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      2162bd8c
  22. 03 May, 2012 5 commits
  23. 22 Jun, 2011 3 commits
    • Jeff King's avatar
      upload-archive: allow user to turn off filters · 7b97730b
      Jeff King authored
      Some tar filters may be very expensive to run, so sites do
      not want to expose them via upload-archive. This patch lets
      users configure tar.<filter>.remote to turn them off.
      
      By default, gzip filters are left on, as they are about as
      expensive as creating zip archives.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      7b97730b
    • Jeff King's avatar
      archive: pass archiver struct to write_archive callback · 4d7c9898
      Jeff King authored
      The current archivers are very static; when you are in the
      write_tar_archive function, you know you are writing a tar.
      However, to facilitate runtime-configurable archivers
      that will share a common write function we need to tell the
      function which archiver was used.
      
      As a convenience, we also provide an opaque data pointer in
      the archiver struct so that individual archivers can put
      something useful there when they register themselves.
      Technically they could just use the "name" field to look in
      an internal map of names to data, but this is much simpler.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      4d7c9898
    • Jeff King's avatar
      archive: refactor list of archive formats · 13e0f88d
      Jeff King authored
      Most of the tar and zip code was nicely split out into two
      abstracted files which knew only about their specific
      formats. The entry point to this code was a single "write
      archive" function.
      
      However, as these basic formats grow more complex (e.g., by
      handling multiple file extensions and format names), a
      static list of the entry point functions won't be enough.
      Instead, let's provide a way for the tar and zip code to
      tell the main archive code what they support by registering
      archiver names and functions.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      13e0f88d
  24. 10 Jun, 2011 1 commit
    • Junio C Hamano's avatar
      zlib: zlib can only process 4GB at a time · ef49a7a0
      Junio C Hamano authored
      The size of objects we read from the repository and data we try to put
      into the repository are represented in "unsigned long", so that on larger
      architectures we can handle objects that weigh more than 4GB.
      
      But the interface defined in zlib.h to communicate with inflate/deflate
      limits avail_in (how many bytes of input are we calling zlib with) and
      avail_out (how many bytes of output from zlib are we ready to accept)
      fields effectively to 4GB by defining their type to be uInt.
      
      In many places in our code, we allocate a large buffer (e.g. mmap'ing a
      large loose object file) and tell zlib its size by assigning the size to
      avail_in field of the stream, but that will truncate the high octets of
      the real size. The worst part of this story is that we often pass around
      z_stream (the state object used by zlib) to keep track of the number of
      used bytes in input/output buffer by inspecting these two fields, which
      practically limits our callchain to the same 4GB limit.
      
      Wrap z_stream in another structure git_zstream that can express avail_in
      and avail_out in unsigned long. For now, just die() when the caller gives
      a size that cannot be given to a single zlib call. In later patches in the
      series, we would make git_inflate() and git_deflate() internally loop to
      give callers an illusion that our "improved" version of zlib interface can
      operate on a buffer larger than 4GB in one go.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      ef49a7a0