1. 16 Jan, 2018 3 commits
  2. 06 Dec, 2017 1 commit
  3. 27 Nov, 2017 1 commit
  4. 09 Nov, 2017 1 commit
  5. 05 Aug, 2016 5 commits
    • Jeff King's avatar
      trace: do not fall back to stderr · 6f253057
      Jeff King authored
      If the trace code cannot open a specified file, or does not
      understand the contents of the GIT_TRACE variable, it falls
      back to printing trace output to stderr.
      This is an attempt to be helpful, but in practice it just
      ends up annoying. The user was trying to get the output to
      go somewhere else, so spewing it to stderr does not really
      accomplish that. And as it's intended for debugging, they
      can presumably re-run the command with their error
      So instead of falling back, this patch disables bogus trace
      keys for the rest of the program, just as we do for write
      errors. We can drop the "Defaulting to..." part of the error
      message entirely; after seeing "cannot open '/foo'", the
      user can assume that tracing is skipped.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Jeff King's avatar
      trace: disable key after write error · 46ac74b7
      Jeff King authored
      If we get a write error writing to a trace descriptor, the
      error isn't likely to go away if we keep writing. Instead,
      you'll just get the same error over and over. E.g., try:
        GIT_TRACE_PACKET=42 git ls-remote >/dev/null
      You don't really need to see:
        warning: unable to write trace for GIT_TRACE_PACKET: Bad file descriptor
      hundreds of times. We could fallback to tracing to stderr,
      as we do in the error code-path for open(), but there's not
      much point. If the user fed us a bogus descriptor, they're
      probably better off fixing their invocation. And if they
      didn't, and we saw a transient error (e.g., ENOSPC writing
      to a file), it probably doesn't help anybody to have half of
      the trace in a file, and half on stderr.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Jeff King's avatar
      trace: correct variable name in write() error message · 3b0c3ab7
      Jeff King authored
      Our error message for write() always mentions GIT_TRACE,
      even though we may be writing for a different variable
      entirely. It's also not quite accurate to say "fd given by
      GIT_TRACE environment variable", as we may hit this error
      based on a filename the user put in the variable (we do
      complain and switch to stderr if the file cannot be opened,
      but it's still possible to hit a write() error on the
      descriptor later).
      So let's fix those things, and switch to our more usual
      "unable to do X: Y" format for the error.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Jeff King's avatar
      trace: cosmetic fixes for error messages · b3a1c5da
      Jeff King authored
      The error messages for the trace code are often multi-line;
      the first line gets a nice "warning:", but the rest are
      left-aligned. Let's give them an indentation to make sure
      they stand out as a unit.
      While we're here, let's also downcase the first letter of
      each error (our usual style), and break up a long line of
      advice (since we're already using multiple lines, one more
      doesn't hurt).
      We also replace "What does 'foo' for GIT_TRACE mean?". While
      cute, it's probably a good idea to give more context, and
      follow our usual styles. So it's now "unknown trace value
      for 'GIT_TRACE': foo".
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Jeff King's avatar
      trace: use warning() for printing trace errors · 38f460ca
      Jeff King authored
      Right now we just fprintf() straight to stderr, which can
      make the output hard to distinguish. It would be helpful to
      give it one of our usual prefixes like "error:", "warning:",
      It doesn't make sense to use error() here, as the trace code
      is "optional" debugging code. If something goes wrong, we
      should warn the user, but saying "error" implies the actual
      git operation had a problem. So warning() is the only sane
      Note that this does end up calling warn_routine() to do the
      formatting. This is probably a good thing, since they are
      clearly trying to hook messages before they make it to
      stderr. However, it also means that in theory somebody who
      tries to trace from their warn_routine() could cause a loop.
      This seems rather unlikely in practice (we've never even
      overridden the default warn_builtin routine before, and
      recent discussions to do so would just install a noop
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  6. 04 Aug, 2016 2 commits
    • Jeff King's avatar
      trace: stop using write_or_whine_pipe() · c0222e76
      Jeff King authored
      The write_or_whine_pipe function does two things:
        1. it checks for EPIPE and converts it into a signal death
        2. it prints a message to stderr on error
      The first thing does not help us, and actively hurts.
      Generally we would simply die from SIGPIPE in this case,
      unless somebody has taken the time to ignore SIGPIPE for the
      whole process.  And if they _did_ do that, it seems rather
      silly for the trace code, which otherwise takes pains to
      continue even in the face of errors (e.g., by not using
      write_or_die!), to take down the whole process for one
      specific type of error.
      Nor does the second thing help us; it just makes it harder
      to write our error message, because we have to feed bits of
      it as an argument to write_or_whine_pipe(). Translators
      never get to see the full message, and it's hard for us to
      customize it.
      Let's switch to just using write_in_full() and writing our
      own error string. For now, the error is identical to what
      write_or_whine_pipe() would say, but now that it's more
      under our control, we can improve it in future patches.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Jeff King's avatar
      trace: handle NULL argument in trace_disable() · c81539b5
      Jeff King authored
      All of the trace functions treat a NULL key as a synonym for
      the default GIT_TRACE key. Except for trace_disable(), which
      will segfault.
      Fortunately, this can't cause any bugs, as the function has
      no callers. But rather than drop it, let's fix the bug, as I
      plan to add a caller.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  7. 25 Sep, 2015 1 commit
    • Jeff King's avatar
      trace: use strbuf for quote_crnl output · 0bb443fd
      Jeff King authored
      When we output GIT_TRACE_SETUP paths, we quote any
      meta-characters. But our buffer to hold the result is only
      PATH_MAX bytes, and we could double the size of the input
      path (if every character needs quoting). We could use a
      2*PATH_MAX buffer, if we assume the input will never be more
      than PATH_MAX. But it's easier still to just switch to a
      strbuf and not worry about whether the input can exceed
      PATH_MAX or not.
      The original copied the "p2" pointer to "p1", advancing
      both. Since this gets rid of "p1", let's also drop "p2",
      whose name is now confusing. We can just advance the
      original "path" pointer.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  8. 16 Jun, 2015 1 commit
    • Jeff King's avatar
      pkt-line: support tracing verbatim pack contents · 32359838
      Jeff King authored
      When debugging the pack protocol, it is sometimes useful to
      store the verbatim pack that we sent or received on the
      wire. Looking at the on-disk result is often not helpful for
      a few reasons:
        1. If the operation is a clone, we destroy the repo on
           failure, leaving nothing on disk.
        2. If the pack is small, we unpack it immediately, and the
           full pack never hits the disk.
        3. If we feed the pack to "index-pack --fix-thin", the
           resulting pack has the extra delta bases added to it.
      We already have a GIT_TRACE_PACKET mechanism for tracing
      packets. Let's extend it with GIT_TRACE_PACKFILE to dump the
      verbatim packfile.
      There are a few other positive fallouts that come from
      rearranging this code:
       - We currently disable the packet trace after seeing the
         PACK header, even though we may get human-readable lines
         on other sidebands; now we include them in the trace.
       - We currently try to print "PACK ..." in the trace to
         indicate that the packfile has started. But because we
         disable packet tracing, we never printed this line. We
         will now do so.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  9. 09 Jan, 2015 1 commit
  10. 12 Dec, 2014 1 commit
  11. 01 Dec, 2014 1 commit
  12. 29 Sep, 2014 1 commit
  13. 08 Sep, 2014 1 commit
  14. 26 Aug, 2014 1 commit
  15. 14 Jul, 2014 9 commits
    • Karsten Blees's avatar
      git: add performance tracing for git's main() function to debug scripts · 578da039
      Karsten Blees authored
      Use trace_performance to measure and print execution time and command line
      arguments of the entire main() function. In constrast to the shell's 'time'
      utility, which measures total time of the parent process, this logs all
      involved git commands recursively. This is particularly useful to debug
      performance issues of scripted commands (i.e. which git commands were
      called with which parameters, and how long did they execute).
      Due to git's deliberate use of exit(), the implementation uses an atexit
      routine rather than just adding trace_performance_since() at the end of
      Usage example: > GIT_TRACE_PERFORMANCE=~/git-trace.log git stash list
      Creates a log file like this:
      23:57:38.638765 trace.c:405 performance: 0.000310107 s: git command: 'git' 'rev-parse' '--git-dir'
      23:57:38.644387 trace.c:405 performance: 0.000261759 s: git command: 'git' 'rev-parse' '--show-toplevel'
      23:57:38.646207 trace.c:405 performance: 0.000304468 s: git command: 'git' 'config' '--get-colorbool' 'color.interactive'
      23:57:38.648491 trace.c:405 performance: 0.000241667 s: git command: 'git' 'config' '--get-color' 'color.interactive.help' 'red bold'
      23:57:38.650465 trace.c:405 performance: 0.000243063 s: git command: 'git' 'config' '--get-color' '' 'reset'
      23:57:38.654850 trace.c:405 performance: 0.025126313 s: git command: 'git' 'stash' 'list'
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: add trace_performance facility to debug performance issues · 09b2c1c7
      Karsten Blees authored
      Add trace_performance and trace_performance_since macros that print a
      duration and an optional printf-formatted text to the file specified in
      environment variable GIT_TRACE_PERFORMANCE.
      These macros, in conjunction with getnanotime(), are intended to simplify
      performance measurements from within the application (i.e. profiling via
      manual instrumentation, rather than using an external profiling tool).
      Unless enabled via GIT_TRACE_PERFORMANCE, these macros have no noticeable
      impact on performance, so that test code for well known time killers may
      be shipped in release builds. Alternatively, a developer could provide an
      additional performance patch (not meant for master) that allows reviewers
      to reproduce performance tests more easily, e.g. on other platforms or
      using their own repositories.
      Usage examples:
      Simple use case (measure one code section):
        uint64_t start = getnanotime();
        /* code section to measure */
        trace_performance_since(start, "foobar");
      Complex use case (measure repetitive code sections):
        uint64_t t = 0;
        for (;;) {
          /* ignore */
          t -= getnanotime();
          /* code section to measure */
          t += getnanotime();
          /* ignore */
        trace_performance(t, "frotz");
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: add high resolution timer function to debug performance issues · 148d6771
      Karsten Blees authored
      Add a getnanotime() function that returns nanoseconds since 01/01/1970 as
      unsigned 64-bit integer (i.e. overflows in july 2554). This is easier to
      work with than e.g. struct timeval or struct timespec. Basing the timer on
      the epoch allows using the results with other time-related APIs.
      To simplify adaption to different platforms, split the implementation into
      a common getnanotime() and a platform-specific highres_nanos() function.
      The common getnanotime() function handles errors, falling back to
      gettimeofday() if highres_nanos() isn't implemented or doesn't work.
      getnanotime() is also responsible for normalizing to the epoch. The offset
      to the system clock is calculated only once on initialization, i.e.
      manually setting the system clock has no impact on the timer (except if
      the fallback gettimeofday() is in use). Git processes are typically short
      lived, so we don't need to handle clock drift.
      The highres_nanos() function returns monotonically increasing nanoseconds
      relative to some arbitrary point in time (e.g. system boot), or 0 on
      failure. Providing platform-specific implementations should be relatively
      easy, e.g. adapting to clock_gettime() as defined by the POSIX realtime
      extensions is seven lines of code.
      This version includes highres_nanos() implementations for:
       * Linux: using clock_gettime(CLOCK_MONOTONIC)
       * Windows: using QueryPerformanceCounter()
       * enable clock_gettime() on more platforms
       * add Mac OSX version, e.g. using mach_absolute_time + mach_timebase_info
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: add 'file:line' to all trace output · e05bed96
      Karsten Blees authored
      This is useful to see where trace output came from.
      Add 'const char *file, int line' parameters to the printing functions and
      rename them to *_fl.
      Add trace_printf* and trace_strbuf macros resolving to the *_fl functions
      and let the preprocessor fill in __FILE__ and __LINE__.
      As the trace_printf* functions take a variable number of arguments, this
      requires variadic macros (i.e. '#define foo(...) foo_impl(__VA_ARGS__)'.
      Though part of C99, it is unclear whether older compilers support this.
      Thus keep the old functions and only enable variadic macros for GNUC and
      MSVC 2005+ (_MSC_VER 1400). This has the nice side effect that the old
      C-style declarations serve as documentation how the macros are to be used.
      Print 'file:line ' as prefix to each trace line. Align the remaining trace
      output at column 40 to accommodate 18 char file names + 4 digit line
      number (currently there are 30 *.c files of length 18 and just 11 of 19).
      Trace output from longer source files (e.g. builtin/receive-pack.c) will
      not be aligned.
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: move code around, in preparation to file:line output · 66f66c59
      Karsten Blees authored
      No functional changes, just move stuff around so that the next patch isn't
      that ugly...
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: add current timestamp to all trace output · b72be02c
      Karsten Blees authored
      This is useful to tell apart trace output of separate test runs.
      It can also be used for basic, coarse-grained performance analysis. Note
      that the accuracy is tainted by writing to the trace file, and you have to
      calculate the deltas yourself (which is next to impossible if multiple
      threads or processes are involved).
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: disable additional trace output for unit tests · 124647c4
      Karsten Blees authored
      Some unit-tests use trace output to verify internal state, and unstable
      output such as timestamps and line numbers are not useful there.
      Disable additional trace output if GIT_TRACE_BARE is set.
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: add infrastructure to augment trace output with additional info · c69dfd24
      Karsten Blees authored
      To be able to add a common prefix or suffix to all trace output (e.g.
      a timestamp or file:line of the caller), factor out common setup and
      cleanup tasks of the trace* functions.
      When adding a common prefix, it makes sense that the output of each trace
      call starts on a new line. Add '\n' in case the caller forgot.
      Note that this explicitly limits trace output to line-by-line, it is no
      longer possible to trace-print just part of a line. Until now, this was
      just an implicit assumption (trace-printing part of a line worked, but
      messed up the trace file if multiple threads or processes were involved).
      Thread-safety / inter-process-safety is also the reason why we need to do
      the prefixing and suffixing in memory rather than issuing multiple write()
      calls. Write_or_whine_pipe() / xwrite() is atomic unless the size exceeds
      MAX_IO_SIZE (8MB, see wrapper.c). In case of trace_strbuf, this costs an
      additional string copy (which should be irrelevant for performance in light
      of actual file IO).
      While we're at it, rename trace_strbuf's 'buf' argument, which suggests
      that the function is modifying the buffer. Trace_strbuf() currently is the
      only trace API that can print arbitrary binary data (without barfing on
      '%' or stopping at '\0'), so 'data' seems more appropriate.
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
    • Karsten Blees's avatar
      trace: improve trace performance · 6aa30857
      Karsten Blees authored
      The trace API currently rechecks the environment variable and reopens the
      trace file on every API call. This has the ugly side effect that errors
      (e.g. file cannot be opened, or the user specified a relative path) are
      also reported on every call. Performance can be improved by about factor
      three by remembering the environment state and keeping the file open.
      Replace the 'const char *key' parameter in the API with a pointer to a
      'struct trace_key' that bundles the environment variable name with
      additional, trace-internal state. Change the call sites of these APIs to
      use a static 'struct trace_key' instead of a string constant.
      In trace.c::get_trace_fd(), save and reuse the file descriptor in 'struct
      Add a 'trace_disable()' API, so that packet_trace() can cleanly disable
      tracing when it encounters packed data (instead of using unsetenv()).
      Signed-off-by: default avatarKarsten Blees <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  16. 17 Jun, 2014 2 commits
  17. 11 Dec, 2013 1 commit
    • Duy Nguyen's avatar
      shallow.c: the 8 steps to select new commits for .git/shallow · 58babfff
      Duy Nguyen authored
      Suppose a fetch or push is requested between two shallow repositories
      (with no history deepening or shortening). A pack that contains
      necessary objects is transferred over together with .git/shallow of
      the sender. The receiver has to determine whether it needs to update
      .git/shallow if new refs needs new shallow comits.
      The rule here is avoid updating .git/shallow by default. But we don't
      want to waste the received pack. If the pack contains two refs, one
      needs new shallow commits installed in .git/shallow and one does not,
      we keep the latter and reject/warn about the former.
      Even if .git/shallow update is allowed, we only add shallow commits
      strictly necessary for the former ref (remember the sender can send
      more shallow commits than necessary) and pay attention not to
      accidentally cut the receiver history short (no history shortening is
      asked for)
      So the steps to figure out what ref need what new shallow commits are:
      1. Split the sender shallow commit list into "ours" and "theirs" list
         by has_sha1_file. Those that exist in current repo in "ours", the
         remaining in "theirs".
      2. Check the receiver .git/shallow, remove from "ours" the ones that
         also exist in .git/shallow.
      3. Fetch the new pack. Either install or unpack it.
      4. Do has_sha1_file on "theirs" list again. Drop the ones that fail
         has_sha1_file. Obviously the new pack does not need them.
      5. If the pack is kept, remove from "ours" the ones that do not exist
         in the new pack.
      6. Walk the new refs to answer the question "what shallow commits,
         both ours and theirs, are required in .git/shallow in order to add
         this ref?". Shallow commits not associated to any refs are removed
         from their respective list.
      7. (*) Check reachability (from the current refs) of all remaining
         commits in "ours". Those reachable are removed. We do not want to
         cut any part of our (reachable) history. We only check up
         commits. True reachability test is done by
         check_everything_connected() at the end as usual.
      8. Combine the final "ours" and "theirs" and add them all to
         .git/shallow. Install new refs. The case where some hook rejects
         some refs on a push is explained in more detail in the push
      Of these steps, #6 and #7 are expensive. Both require walking through
      some commits, or in the worst case all commits. And we rather avoid
      them in at least common case, where the transferred pack does not
      contain any shallow commits that the sender advertises. Let's look at
      each scenario:
      1) the sender has longer history than the receiver
         All shallow commits from the sender will be put into "theirs" list
         at step 1 because none of them exists in current repo. In the
         common case, "theirs" becomes empty at step 4 and exit early.
      2) the sender has shorter history than the receiver
         All shallow commits from the sender are likely in "ours" list at
         step 1. In the common case, if the new pack is kept, we could empty
         "ours" and exit early at step 5.
         If the pack is not kept, we hit the expensive step 6 then exit
         after "ours" is emptied. There'll be only a handful of objects to
         walk in fast-forward case. If it's forced update, we may need to
         walk to the bottom.
      3) the sender has same .git/shallow as the receiver
         This is similar to case 2 except that "ours" should be emptied at
         step 2 and exit early.
      A fetch after "clone --depth=X" is case 1. A fetch after "clone" (from
      a shallow repo) is case 3. Luckily they're cheap for the common case.
      A push from "clone --depth=X" falls into case 2, which is expensive.
      Some more work may be done at the sender/client side to avoid more
      work on the server side: if the transferred pack does not contain any
      shallow commits, send-pack should not send any shallow commits to the
      receive-pack, effectively turning it into a normal push and avoid all
      This patch implements all steps except #3, already handled by
      fetch-pack and receive-pack, #6 and #7, which has their own patch due
      to their size.
      (*) in previous versions step 7 was put before step 3. I reorder it so
          that the common case that keeps the pack does not need to walk
          commits at all. In future if we implement faster commit
          reachability check (maybe with the help of pack bitmaps or commit
          cache), step 7 could become cheap and be moved up before 6 again.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  18. 10 Jul, 2013 1 commit
    • Jeff King's avatar
      add missing "format" function attributes · 4621085b
      Jeff King authored
      For most of our functions that take printf-like formats, we
      use gcc's __attribute__((format)) to get compiler warnings
      when the functions are misused. Let's give a few more
      functions the same protection.
      In most cases, the annotations do not uncover any actual
      bugs; the only code change needed is that we passed a size_t
      to transfer_debug, which expected an int. Since we expect
      the passed-in value to be a relatively small buffer size
      (and cast a similar value to int directly below), we can
      just cast away the problem.
      Signed-off-by: default avatarJeff King <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  19. 16 Sep, 2012 1 commit
  20. 22 Mar, 2011 1 commit
    • Stephen Boyd's avatar
      Fix sparse warnings · c2e86add
      Stephen Boyd authored
      Fix warnings from 'make check'.
       - These files don't include 'builtin.h' causing sparse to complain that
         cmd_* isn't declared:
         builtin/clone.c:364, builtin/fetch-pack.c:797,
         builtin/fmt-merge-msg.c:34, builtin/hash-object.c:78,
         builtin/merge-index.c:69, builtin/merge-recursive.c:22
         builtin/merge-tree.c:341, builtin/mktag.c:156, builtin/notes.c:426
         builtin/notes.c:822, builtin/pack-redundant.c:596,
         builtin/pack-refs.c:10, builtin/patch-id.c:60, builtin/patch-id.c:149,
         builtin/remote.c:1512, builtin/remote-ext.c:240,
         builtin/remote-fd.c:53, builtin/reset.c:236, builtin/send-pack.c:384,
         builtin/unpack-file.c:25, builtin/var.c:75
       - These files have symbols which should be marked static since they're
         only file scope:
         submodule.c:12, diff.c:631, replace_object.c:92, submodule.c:13,
         submodule.c:14, trace.c:78, transport.c:195, transport-helper.c:79,
         unpack-trees.c:19, url.c:3, url.c:18, url.c:104, url.c:117, url.c:123,
         url.c:129, url.c:136, thread-utils.c:21, thread-utils.c:48
       - These files redeclare symbols to be different types:
         builtin/index-pack.c:210, parse-options.c:564, parse-options.c:571,
         usage.c:49, usage.c:58, usage.c:63, usage.c:72
       - These files use a literal integer 0 when they really should use a NULL
         daemon.c:663, fast-import.c:2942, imap-send.c:1072, notes-merge.c:362
      While we're in the area, clean up some unused #includes in builtin files
      (mostly exec_cmd.h).
      Signed-off-by: default avatarStephen Boyd <[email protected]>
      Signed-off-by: default avatarJunio C Hamano <[email protected]>
  21. 08 Mar, 2011 4 commits