1. 16 Apr, 2018 1 commit
    • Duy Nguyen's avatar
      gc --auto: exclude base pack if not enough mem to "repack -ad" · 9806f5a7
      Duy Nguyen authored
      pack-objects could be a big memory hog especially on large repos,
      everybody knows that. The suggestion to stick a .keep file on the
      giant base pack to avoid this problem is also known for a long time.
      
      Recent patches add an option to do just this, but it has to be either
      configured or activated manually. This patch lets `git gc --auto`
      activate this mode automatically when it thinks `repack -ad` will use
      a lot of memory and start affecting the system due to swapping or
      flushing OS cache.
      
      gc --auto decides to do this based on an estimation of pack-objects
      memory usage, which is quite accurate at least for the heap part, and
      whether that fits in half of system memory (the assumption here is for
      desktop environment where there are many other applications running).
      
      This mechanism only kicks in if gc.bigBasePackThreshold is not configured.
      If it is, it is assumed that the user already knows what they want.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      9806f5a7
  2. 11 Apr, 2018 2 commits
  3. 12 Feb, 2018 1 commit
  4. 25 Sep, 2017 1 commit
    • Michael Haggerty's avatar
      packed_ref_cache: keep the `packed-refs` file mmapped if possible · 5b633610
      Michael Haggerty authored
      Keep a copy of the `packed-refs` file contents in memory for as long
      as a `packed_ref_cache` object is in use:
      
      * If the system allows it, keep the `packed-refs` file mmapped.
      
      * If not (either because the system doesn't support `mmap()` at all,
        or because a file that is currently mmapped cannot be replaced via
        `rename()`), then make a copy of the file's contents in
        heap-allocated space, and keep that around instead.
      
      We base the choice of behavior on a new build-time switch,
      `MMAP_PREVENTS_DELETE`. By default, this switch is set for Windows
      variants.
      
      After this commit, `MMAP_NONE` and `MMAP_TEMPORARY` are still handled
      identically. But the next commit will introduce a difference.
      
      This whole change is still pointless, because we only read the
      `packed-refs` file contents immediately after instantiating the
      `packed_ref_cache`. But that will soon change.
      Signed-off-by: default avatarMichael Haggerty <mhagger@alum.mit.edu>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      5b633610
  5. 21 Jul, 2017 1 commit
  6. 05 Jul, 2017 1 commit
    • Torsten Bögershausen's avatar
      cygwin: allow pushing to UNC paths · 496f2569
      Torsten Bögershausen authored
       cygwin can use an UNC path like //server/share/repo
      
       $ cd //server/share/dir
       $ mkdir test
       $ cd test
       $ git init --bare
      
       However, when we try to push from a local Git repository to this repo,
       there is a problem: Git converts the leading "//" into a single "/".
      
       As cygwin handles an UNC path so well, Git can support them better:
      
       - Introduce cygwin_offset_1st_component() which keeps the leading "//",
         similar to what Git for Windows does.
      
       - Move CYGWIN out of the POSIX in the tests for path normalization in t0060
      Signed-off-by: default avatarTorsten Bögershausen <tboegi@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      496f2569
  7. 01 Jun, 2017 1 commit
    • Ævar Arnfjörð Bjarmason's avatar
      grep: un-break building with PCRE >= 8.32 without --enable-jit · fb95e2e3
      Ævar Arnfjörð Bjarmason authored
      Amend my change earlier in this series ("grep: add support for the
      PCRE v1 JIT API", 2017-04-11) to un-break the build on PCRE v1
      versions later than 8.31 compiled without --enable-jit.
      
      As explained in that change and a later compatibility change in this
      series ("grep: un-break building with PCRE < 8.32", 2017-05-10) the
      pcre_jit_exec() function is a faster path to execute the JIT.
      
      Unfortunately there's no compatibility stub for that function compiled
      into the library if pcre_config(PCRE_CONFIG_JIT, &ret) would return 0,
      and no macro that can be used to check for it, so the only portable
      option to support builds without --enable-jit is via a new
      NO_LIBPCRE1_JIT=UnfortunatelyYes Makefile option[1].
      
      Another option would be to make the JIT opt-in via
      USE_LIBPCRE1_JIT=YesPlease, after all it's not a default option of
      PCRE v1.
      
      I think it makes more sense to make it opt-out since even though it's
      not a default option, most packagers of PCRE seem to turn it on by
      default, with the notable exception of the MinGW package.
      
      Make the MinGW platform work by default by changing the build defaults
      to turn on NO_LIBPCRE1_JIT=UnfortunatelyYes. It is the only platform
      that turns on USE_LIBPCRE=YesPlease by default, see commit
      df5218b4 ("config.mak.uname: support MSys2", 2016-01-13) for that
      change.
      
      1. "How do I support pcre1 JIT on all
         versions?"  (https://lists.exim.org/lurker/thread/20170601.103148.10253788.en.html)
      
      2. https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-pcre/PKGBUILD
         (referenced from "Re: PCRE v2 compile error, was Re: What's cooking
         in git.git (May 2017, #01; Mon, 1)";
         <alpine.DEB.2.20.1705021756530.3480@virtualbox>)
      Signed-off-by: Ævar Arnfjörð Bjarmason's avatarÆvar Arnfjörð Bjarmason <avarab@gmail.com>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      fb95e2e3
  8. 26 May, 2017 2 commits
  9. 04 May, 2017 1 commit
  10. 28 Feb, 2017 1 commit
  11. 09 Feb, 2017 1 commit
    • Jeff Hostetler's avatar
      mingw: use OpenSSL's SHA-1 routines · 2cfc70f0
      Jeff Hostetler authored
      Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
      This improves performance on SHA1 operations on Intel processors.
      
      OpenSSL 1.0.2 has made considerable performance improvements and
      support the Intel hardware acceleration features.  See:
      https://software.intel.com/en-us/articles/improving-openssl-performance
      https://software.intel.com/en-us/articles/intel-sha-extensions
      
      To test this I added/staged a single file in a gigantic
      repository having a 450MB index file.  The code in read-cache.c
      verifies the header SHA as it reads the index and computes a new
      header SHA as it writes out the new index.  Therefore, in this test
      the SHA code must process 900MB of data.  Testing was done on an
      Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU.
      
      The block-sha1 version averaged 5.27 seconds.
      The OpenSSL    version averaged 4.50 seconds.
      
      ================================================================
      
      $ echo xxx >> project.mk
      $ time /e/blk_sha/bin/git.exe add project.mk
      
      real    0m5.207s
      user    0m0.000s
      sys     0m0.250s
      
      $ echo xxx >> project.mk
      $ time /e/blk_sha/bin/git.exe add project.mk
      
      real    0m5.362s
      user    0m0.015s
      sys     0m0.234s
      
      $ echo xxx >> project.mk
      $ time /e/blk_sha/bin/git.exe add project.mk
      
      real    0m5.300s
      user    0m0.016s
      sys     0m0.250s
      
      $ echo xxx >> project.mk
      $ time /e/blk_sha/bin/git.exe add project.mk
      
      real    0m5.216s
      user    0m0.000s
      sys     0m0.250s
      
      ================================================================
      $ echo xxx >> project.mk
      $ time /e/openssl/bin/git.exe add project.mk
      
      real    0m4.431s
      user    0m0.000s
      sys     0m0.250s
      
      $ echo xxx >> project.mk
      $ time /e/openssl/bin/git.exe add project.mk
      
      real    0m4.478s
      user    0m0.000s
      sys     0m0.265s
      
      $ echo xxx >> project.mk
      $ time /e/openssl/bin/git.exe add project.mk
      
      real    0m4.690s
      user    0m0.000s
      sys     0m0.250s
      
      $ echo xxx >> project.mk
      $ time /e/openssl/bin/git.exe add project.mk
      
      real    0m4.420s
      user    0m0.000s
      sys     0m0.234s
      
      ================================================================
      Signed-off-by: default avatarJeff Hostetler <jeffhost@microsoft.com>
      Signed-off-by: Johannes Schindelin's avatarJohannes Schindelin <johannes.schindelin@gmx.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      2cfc70f0
  12. 06 Dec, 2016 1 commit
    • Jeff King's avatar
      xdiff: drop XDL_FAST_HASH · 1f7c9261
      Jeff King authored
      The xdiff code hashes every line of both sides of a diff,
      and then compares those hashes to find duplicates. The
      overall performance depends both on how fast we can compute
      the hashes, but also on how many hash collisions we see.
      
      The idea of XDL_FAST_HASH is to speed up the hash
      computation. But the generated hashes have worse collision
      behavior. This means that in some cases it speeds diffs up
      (running "git log -p" on git.git improves by ~8% with it),
      but in others it can slow things down. One pathological case
      saw over a 100x slowdown[1].
      
      There may be a better hash function that covers both
      properties, but in the meantime we are better off with the
      original hash. It's slightly slower in the common case, but
      it has fewer surprising pathological cases.
      
      [1] http://public-inbox.org/git/20141222041944.GA441@peff.net/Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      1f7c9261
  13. 04 Aug, 2016 1 commit
  14. 26 Jul, 2016 1 commit
  15. 11 Jul, 2016 1 commit
  16. 26 May, 2016 1 commit
    • Karsten Blees's avatar
      mingw: make isatty() recognize MSYS2's pseudo terminals (/dev/pty*) · f7f90e0f
      Karsten Blees authored
      MSYS2 emulates pseudo terminals via named pipes, and isatty() returns 0
      for such file descriptors. Therefore, some interactive functionality
      (such as launching a pager, asking if a failed unlink should be repeated
      etc.) doesn't work when run in a terminal emulator that uses MSYS2's
      ptys (such as mintty).
      
      However, MSYS2 uses special names for its pty pipes ('msys-*-pty*'),
      which allows us to distinguish them from normal piped input / output.
      
      On startup, check if stdin / stdout / stderr are connected to such pipes
      using the NtQueryObject API from NTDll.dll. If the names match, adjust
      the flags in MSVCRT's ioinfo structure accordingly.
      Signed-off-by: default avatarKarsten Blees <blees@dcon.de>
      Signed-off-by: Johannes Schindelin's avatarJohannes Schindelin <johannes.schindelin@gmx.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      f7f90e0f
  17. 21 Mar, 2016 1 commit
  18. 10 Mar, 2016 1 commit
    • Junio C Hamano's avatar
      sane_grep: pass "-a" if grep accepts it · 71b40103
      Junio C Hamano authored
      Newer versions of GNU grep is reported to be pickier when we feed a
      non-ASCII input and break some Porcelain scripts.  As we know we do
      not feed random binary file to our own sane_grep wrapper, allow us
      to always pass "-a" by setting SANE_TEXT_GREP=-a Makefile variable
      to work it around.
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      71b40103
  19. 29 Feb, 2016 1 commit
    • Torsten Bögershausen's avatar
      config.mak.uname: use clang for Mac OS X 10.6 · 7b6daf8d
      Torsten Bögershausen authored
      Gcc under Mac OX 10.6 throws an internal compiler error:
      
      CC combine-diff.o
          combine-diff.c: In function ‘diff_tree_combined’:
          combine-diff.c:1391: internal compiler error: Segmentation fault
      
      while attempting to build Git at 5b442c4f (tree-diff: catch integer
      overflow in combine_diff_path allocation, 2016-02-19).
      
      As clang that ships with the version does not have the same bug,
      make Git compile under Mac OS X 10.6 by using clang instead of gcc
      to work this around, as it is unlikely that we will see fixed GCC
      on that platform.
      
      Later versions of Mac OSX/Xcode only provide clang, and gcc is a
      wrapper to it.
      Signed-off-by: default avatarTorsten Bögershausen <tboegi@web.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      7b6daf8d
  20. 26 Jan, 2016 2 commits
  21. 13 Jan, 2016 2 commits
    • Johannes Schindelin's avatar
      config.mak.uname: supporting 64-bit MSys2 · 7b40ae86
      Johannes Schindelin authored
      This just makes things compile, the test suite needs extra tender loving
      care in addition to this change. We will address these issues in later
      commits.
      
      While at it, also allow building MSys2 Git (i.e. a Git that uses MSys2's
      POSIX emulation layer).
      Signed-off-by: Johannes Schindelin's avatarJohannes Schindelin <johannes.schindelin@gmx.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      7b40ae86
    • Johannes Schindelin's avatar
      config.mak.uname: support MSys2 · df5218b4
      Johannes Schindelin authored
      For a long time, Git for Windows lagged behind Git's 2.x releases because
      the Git for Windows developers wanted to let that big jump coincide with
      a well-needed jump away from MSys to MSys2.
      
      To understand why this is such a big issue, it needs to be noted that
      many parts of Git are not written in portable C, but instead Git relies
      on a POSIX shell and Perl to be available.
      
      To support the scripts, Git for Windows has to ship a minimal POSIX
      emulation layer with Bash and Perl thrown in, and when the Git for
      Windows effort started in August 2007, this developer settled on using
      MSys, a stripped down version of Cygwin. Consequently, the original name
      of the project was "msysGit" (which, sadly, caused a *lot* of confusion
      because few Windows users know about MSys, and even less care).
      
      To compile the C code of Git for Windows, MSys was used, too: it sports
      two versions of the GNU C Compiler: one that links implicitly to the
      POSIX emulation layer, and another one that targets the plain Win32 API
      (with a few convenience functions thrown in).  Git for Windows'
      executables are built using the latter, and therefore they are really
      just Win32 programs. To discern executables requiring the POSIX
      emulation layer from the ones that do not, the latter are called MinGW
      (Minimal GNU for Windows) when the former are called MSys executables.
      
      This reliance on MSys incurred challenges, too, though: some of our
      changes to the MSys runtime -- necessary to support Git for Windows
      better -- were not accepted upstream, so we had to maintain our own
      fork. Also, the MSys runtime was not developed further to support e.g.
      UTF-8 or 64-bit, and apart from lacking a package management system
      until much later (when mingw-get was introduced), many packages provided
      by the MSys/MinGW project lag behind the respective source code
      versions, in particular Bash and OpenSSL. For a while, the Git for
      Windows project tried to remedy the situation by trying to build newer
      versions of those packages, but the situation quickly became untenable,
      especially with problems like the Heartbleed bug requiring swift action
      that has nothing to do with developing Git for Windows further.
      
      Happily, in the meantime the MSys2 project (https://msys2.github.io/)
      emerged, and was chosen to be the base of the Git for Windows 2.x. Just
      like MSys, MSys2 is a stripped down version of Cygwin, but it is
      actively kept up-to-date with Cygwin's source code.  Thereby, it already
      supports Unicode internally, and it also offers the 64-bit support that
      we yearned for since the beginning of the Git for Windows project.
      
      MSys2 also ported the Pacman package management system from Arch Linux
      and uses it heavily. This brings the same convenience to which Linux
      users are used to from `yum` or `apt-get`, and to which MacOSX users are
      used to from Homebrew or MacPorts, or BSD users from the Ports system,
      to MSys2: a simple `pacman -Syu` will update all installed packages to
      the newest versions currently available.
      
      MSys2 is also *very* active, typically providing package updates
      multiple times per week.
      
      It still required a two-month effort to bring everything to a state
      where Git's test suite passes, many more months until the first official
      Git for Windows 2.x was released, and a couple of patches still await
      their submission to the respective upstream projects. Yet without MSys2,
      the modernization of Git for Windows would simply not have happened.
      
      This commit lays the ground work to supporting MSys2-based Git builds.
      Assisted-by: default avatarWaldek Maleska <weakcamel@users.github.com>
      Signed-off-by: Johannes Schindelin's avatarJohannes Schindelin <johannes.schindelin@gmx.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      df5218b4
  22. 05 Oct, 2015 1 commit
  23. 07 Aug, 2015 1 commit
  24. 03 Jun, 2015 1 commit
  25. 16 Apr, 2015 1 commit
    • Jeff King's avatar
      strbuf_getwholeline: use getdelim if it is available · 0cc30e0e
      Jeff King authored
      We spend a lot of time in strbuf_getwholeline in a tight
      loop reading characters from a stdio handle into a buffer.
      The libc getdelim() function can do this for us with less
      overhead. It's in POSIX.1-2008, and was a GNU extension
      before that. Therefore we can't rely on it, but can fall
      back to the existing getc loop when it is not available.
      
      The HAVE_GETDELIM knob is turned on automatically for Linux,
      where we have glibc. We don't need to set any new
      feature-test macros, because we already define _GNU_SOURCE.
      Other systems that implement getdelim may need to other
      macros (probably _POSIX_C_SOURCE >= 200809L), but we can
      address that along with setting the Makefile knob after
      testing the feature on those systems.
      
      Running "git rev-parse refs/heads/does-not-exist" on a repo
      with an extremely large (1.6GB) packed-refs file went from
      (best-of-5):
      
        real    0m8.601s
        user    0m8.084s
        sys     0m0.524s
      
      to:
      
        real    0m6.768s
        user    0m6.340s
        sys     0m0.432s
      
      for a wall-clock speedup of 21%.
      
      Based on a patch from Rasmus Villemoes <rv@rasmusvillemoes.dk>.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      0cc30e0e
  26. 10 Mar, 2015 1 commit
  27. 09 Jan, 2015 1 commit
  28. 17 Dec, 2014 2 commits
    • Johannes Schindelin's avatar
      read-cache: optionally disallow NTFS .git variants · 2b4c6efc
      Johannes Schindelin authored
      The point of disallowing ".git" in the index is that we
      would never want to accidentally overwrite files in the
      repository directory. But this means we need to respect the
      filesystem's idea of when two paths are equal. The prior
      commit added a helper to make such a comparison for NTFS
      and FAT32; let's use it in verify_path().
      
      We make this check optional for two reasons:
      
        1. It restricts the set of allowable filenames, which is
           unnecessary for people who are not on NTFS nor FAT32.
           In practice this probably doesn't matter, though, as
           the restricted names are rather obscure and almost
           certainly would never come up in practice.
      
        2. It has a minor performance penalty for every path we
           insert into the index.
      
      This patch ties the check to the core.protectNTFS config
      option. Though this is expected to be most useful on Windows,
      we allow it to be set everywhere, as NTFS may be mounted on
      other platforms. The variable does default to on for Windows,
      though.
      Signed-off-by: Johannes Schindelin's avatarJohannes Schindelin <johannes.schindelin@gmx.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      2b4c6efc
    • Jeff King's avatar
      read-cache: optionally disallow HFS+ .git variants · a42643aa
      Jeff King authored
      The point of disallowing ".git" in the index is that we
      would never want to accidentally overwrite files in the
      repository directory. But this means we need to respect the
      filesystem's idea of when two paths are equal. The prior
      commit added a helper to make such a comparison for HFS+;
      let's use it in verify_path.
      
      We make this check optional for two reasons:
      
        1. It restricts the set of allowable filenames, which is
           unnecessary for people who are not on HFS+. In practice
           this probably doesn't matter, though, as the restricted
           names are rather obscure and almost certainly would
           never come up in practice.
      
        2. It has a minor performance penalty for every path we
           insert into the index.
      
      This patch ties the check to the core.protectHFS config
      option. Though this is expected to be most useful on OS X,
      we allow it to be set everywhere, as HFS+ may be mounted on
      other platforms. The variable does default to on for OS X,
      though.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      a42643aa
  29. 15 Aug, 2014 2 commits
  30. 21 Jul, 2014 1 commit
  31. 14 Jul, 2014 1 commit
    • Karsten Blees's avatar
      trace: add high resolution timer function to debug performance issues · 148d6771
      Karsten Blees authored
      Add a getnanotime() function that returns nanoseconds since 01/01/1970 as
      unsigned 64-bit integer (i.e. overflows in july 2554). This is easier to
      work with than e.g. struct timeval or struct timespec. Basing the timer on
      the epoch allows using the results with other time-related APIs.
      
      To simplify adaption to different platforms, split the implementation into
      a common getnanotime() and a platform-specific highres_nanos() function.
      
      The common getnanotime() function handles errors, falling back to
      gettimeofday() if highres_nanos() isn't implemented or doesn't work.
      
      getnanotime() is also responsible for normalizing to the epoch. The offset
      to the system clock is calculated only once on initialization, i.e.
      manually setting the system clock has no impact on the timer (except if
      the fallback gettimeofday() is in use). Git processes are typically short
      lived, so we don't need to handle clock drift.
      
      The highres_nanos() function returns monotonically increasing nanoseconds
      relative to some arbitrary point in time (e.g. system boot), or 0 on
      failure. Providing platform-specific implementations should be relatively
      easy, e.g. adapting to clock_gettime() as defined by the POSIX realtime
      extensions is seven lines of code.
      
      This version includes highres_nanos() implementations for:
       * Linux: using clock_gettime(CLOCK_MONOTONIC)
       * Windows: using QueryPerformanceCounter()
      
      Todo:
       * enable clock_gettime() on more platforms
       * add Mac OSX version, e.g. using mach_absolute_time + mach_timebase_info
      Signed-off-by: default avatarKarsten Blees <blees@dcon.de>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      148d6771
  32. 09 Jun, 2014 1 commit
  33. 06 May, 2014 1 commit
  34. 16 Apr, 2014 1 commit
    • Duy Nguyen's avatar
      index-pack: work around thread-unsafe pread() · 39539495
      Duy Nguyen authored
      Multi-threaing of index-pack was disabled with c0f86547
      (index-pack: Disable threading on cygwin - 2012-06-26), because
      pread() implementations for Cygwin and MSYS were not thread
      safe.  Recent Cygwin does offer usable pread() and we enabled
      multi-threading with 103d530f (Cygwin 1.7 has thread-safe pread,
      2013-07-19).
      
      Work around this problem on platforms with a thread-unsafe
      pread() emulation by opening one file handle per thread; it
      would prevent parallel pread() on different file handles from
      stepping on each other.
      
      Also remove NO_THREAD_SAFE_PREAD that was introduced in c0f86547
      because it's no longer used anywhere.
      
      This workaround is unconditional, even for platforms with
      thread-safe pread() because the overhead is small (a couple file
      handles more) and not worth fragmenting the code.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Tested-by: default avatarJohannes Sixt <j6t@kdbg.org>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      39539495