1. 16 Apr, 2018 1 commit
    • Duy Nguyen's avatar
      pack-objects: turn type and in_pack_type to bitfields · fd9b1bae
      Duy Nguyen authored
      An extra field type_valid is added to carry the equivalent of OBJ_BAD
      in the original "type" field. in_pack_type always contains a valid
      type so we only need 3 bits for it.
      
      A note about accepting OBJ_NONE as "valid" type. The function
      read_object_list_from_stdin() can pass this value [1] and it
      eventually calls create_object_entry() where current code skip setting
      "type" field if the incoming type is zero. This does not have any bad
      side effects because "type" field should be memset()'d anyway.
      
      But since we also need to set type_valid now, skipping oe_set_type()
      leaves type_valid zero/false, which will make oe_type() return
      OBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in
      prepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger
      
          fatal: unable to get type of object ...
      
      Accepting OBJ_NONE [2] does sound wrong, but this is how it is has
      been for a very long time and I haven't time to dig in further.
      
      [1] See 5c49c116 (pack-objects: better check_object() performances -
          2007-04-16)
      
      [2] 21666f1a (convert object type handling from a string to a number
          - 2007-02-26)
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      fd9b1bae
  2. 14 Mar, 2018 1 commit
    • brian m. carlson's avatar
      sha1_file: convert sha1_object_info* to object_id · abef9020
      brian m. carlson authored
      Convert sha1_object_info and sha1_object_info_extended to take pointers
      to struct object_id and rename them to use "oid" instead of "sha1" in
      their names.  Update the declaration and definition and apply the
      following semantic patch, plus the standard object_id transforms:
      
      @@
      expression E1, E2;
      @@
      - sha1_object_info(E1.hash, E2)
      + oid_object_info(&E1, E2)
      
      @@
      expression E1, E2;
      @@
      - sha1_object_info(E1->hash, E2)
      + oid_object_info(E1, E2)
      
      @@
      expression E1, E2, E3;
      @@
      - sha1_object_info_extended(E1.hash, E2, E3)
      + oid_object_info_extended(&E1, E2, E3)
      
      @@
      expression E1, E2, E3;
      @@
      - sha1_object_info_extended(E1->hash, E2, E3)
      + oid_object_info_extended(E1, E2, E3)
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      abef9020
  3. 02 Feb, 2018 1 commit
  4. 24 Sep, 2017 1 commit
  5. 08 May, 2017 1 commit
  6. 28 Mar, 2017 2 commits
    • Jeff King's avatar
      odb_mkstemp: write filename into strbuf · 594fa999
      Jeff King authored
      The odb_mkstemp() function expects the caller to provide a
      fixed buffer to write the resulting tempfile name into. But
      it creates the template using snprintf without checking the
      return value. This means we could silently truncate the
      filename.
      
      In practice, it's unlikely that the truncation would end in
      the template-pattern that mkstemp needs to open the file. So
      we'd probably end up failing either way, unless the path was
      specially crafted.
      
      The simplest fix would be to notice the truncation and die.
      However, we can observe that most callers immediately
      xstrdup() the result anyway. So instead, let's switch to
      using a strbuf, which is easier for them (and isn't a big
      deal for the other 2 callers, who can just strbuf_release
      when they're done with it).
      
      Note that many of the callers used static buffers, but this
      was purely to avoid putting a large buffer on the stack. We
      never passed the static buffers out of the function, so
      there's no complicated memory handling we need to change.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      594fa999
    • Jeff King's avatar
      do not check odb_mkstemp return value for errors · 892e723a
      Jeff King authored
      The odb_mkstemp function does not return an error; it dies
      on failure instead. But many of its callers compare the
      resulting descriptor against -1 and die themselves.
      
      Mostly this is just pointless, but it does raise a question
      when looking at the callers: if they show the results of the
      "template" buffer after a failure, what's in it? The answer
      is: it doesn't matter, because it cannot happen.
      
      So let's make that clear by removing the bogus error checks.
      In bitmap_writer_finish(), we can drop the error-handling
      code entirely. In the other two cases, it's shared with the
      open() in another code path; we can just move the
      error-check next to that open() call.
      
      And while we're at it, let's flesh out the function's
      docstring a bit to make the error behavior clear.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      892e723a
  7. 29 Sep, 2016 1 commit
  8. 16 Mar, 2016 2 commits
    • Jeff King's avatar
      list-objects: pass full pathname to callbacks · 2824e184
      Jeff King authored
      When we find a blob at "a/b/c", we currently pass this to
      our show_object_fn callbacks as two components: "a/b/" and
      "c". Callbacks which want the full value then call
      path_name(), which concatenates the two. But this is an
      inefficient interface; the path is a strbuf, and we could
      simply append "c" to it temporarily, then roll back the
      length, without creating a new copy.
      
      So we could improve this by teaching the callsites of
      path_name() this trick (and there are only 3). But we can
      also notice that no callback actually cares about the
      broken-down representation, and simply pass each callback
      the full path "a/b/c" as a string. The callback code becomes
      even simpler, then, as we do not have to worry about freeing
      an allocated buffer, nor rolling back our modification to
      the strbuf.
      
      This is theoretically less efficient, as some callbacks
      would not bother to format the final path component. But in
      practice this is not measurable. Since we use the same
      strbuf over and over, our work to grow it is amortized, and
      we really only pay to memcpy a few bytes.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      2824e184
    • Jeff King's avatar
      list-objects: drop name_path entirely · dc06dc88
      Jeff King authored
      In the previous commit, we left name_path as a thin wrapper
      around a strbuf. This patch drops it entirely. As a result,
      every show_object_fn callback needs to be adjusted. However,
      none of their code needs to be changed at all, because the
      only use was to pass it to path_name(), which now handles
      the bare strbuf.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      dc06dc88
  9. 12 Feb, 2016 2 commits
    • Jeff King's avatar
      list-objects: pass full pathname to callbacks · de1e67d0
      Jeff King authored
      When we find a blob at "a/b/c", we currently pass this to
      our show_object_fn callbacks as two components: "a/b/" and
      "c". Callbacks which want the full value then call
      path_name(), which concatenates the two. But this is an
      inefficient interface; the path is a strbuf, and we could
      simply append "c" to it temporarily, then roll back the
      length, without creating a new copy.
      
      So we could improve this by teaching the callsites of
      path_name() this trick (and there are only 3). But we can
      also notice that no callback actually cares about the
      broken-down representation, and simply pass each callback
      the full path "a/b/c" as a string. The callback code becomes
      even simpler, then, as we do not have to worry about freeing
      an allocated buffer, nor rolling back our modification to
      the strbuf.
      
      This is theoretically less efficient, as some callbacks
      would not bother to format the final path component. But in
      practice this is not measurable. Since we use the same
      strbuf over and over, our work to grow it is amortized, and
      we really only pay to memcpy a few bytes.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      de1e67d0
    • Jeff King's avatar
      list-objects: drop name_path entirely · bd64516a
      Jeff King authored
      In the previous commit, we left name_path as a thin wrapper
      around a strbuf. This patch drops it entirely. As a result,
      every show_object_fn callback needs to be adjusted. However,
      none of their code needs to be changed at all, because the
      only use was to pass it to path_name(), which now handles
      the bare strbuf.
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      bd64516a
  10. 20 Nov, 2015 3 commits
  11. 01 Dec, 2014 1 commit
    • Karsten Blees's avatar
      pack-bitmap: do not use gcc packed attribute · b5007211
      Karsten Blees authored
      The "__attribute__" flag may be a noop on some compilers.
      That's OK as long as the code is correct without the
      attribute, but in this case it is not. We would typically
      end up with a struct that is 2 bytes too long due to struct
      padding, breaking both reading and writing of bitmaps.
      
      Instead of marshalling the data in a struct, let's just
      provide helpers for reading and writing the appropriate
      types. Besides being correct on all platforms, the result is
      more efficient and simpler to read.
      Signed-off-by: default avatarKarsten Blees <blees@dcon.de>
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      b5007211
  12. 18 Sep, 2014 1 commit
  13. 06 Mar, 2014 1 commit
  14. 30 Dec, 2013 2 commits
    • Vicent Marti's avatar
      pack-bitmap: implement optional name_hash cache · ae4f07fb
      Vicent Marti authored
      When we use pack bitmaps rather than walking the object
      graph, we end up with the list of objects to include in the
      packfile, but we do not know the path at which any tree or
      blob objects would be found.
      
      In a recently packed repository, this is fine. A fetch would
      use the paths only as a heuristic in the delta compression
      phase, and a fully packed repository should not need to do
      much delta compression.
      
      As time passes, though, we may acquire more objects on top
      of our large bitmapped pack. If clients fetch frequently,
      then they never even look at the bitmapped history, and all
      works as usual. However, a client who has not fetched since
      the last bitmap repack will have "have" tips in the
      bitmapped history, but "want" newer objects.
      
      The bitmaps themselves degrade gracefully in this
      circumstance. We manually walk the more recent bits of
      history, and then use bitmaps when we hit them.
      
      But we would also like to perform delta compression between
      the newer objects and the bitmapped objects (both to delta
      against what we know the user already has, but also between
      "new" and "old" objects that the user is fetching). The lack
      of pathnames makes our delta heuristics much less effective.
      
      This patch adds an optional cache of the 32-bit name_hash
      values to the end of the bitmap file. If present, a reader
      can use it to match bitmapped and non-bitmapped names during
      delta compression.
      
      Here are perf results for p5310:
      
      Test                      origin/master       HEAD^                      HEAD
      -------------------------------------------------------------------------------------------------
      5310.2: repack to disk    36.81(37.82+1.43)   47.70(48.74+1.41) +29.6%   47.75(48.70+1.51) +29.7%
      5310.3: simulated clone   30.78(29.70+2.14)   1.08(0.97+0.10) -96.5%     1.07(0.94+0.12) -96.5%
      5310.4: simulated fetch   3.16(6.10+0.08)     3.54(10.65+0.06) +12.0%    1.70(3.07+0.06) -46.2%
      5310.6: partial bitmap    36.76(43.19+1.81)   6.71(11.25+0.76) -81.7%    4.08(6.26+0.46) -88.9%
      
      You can see that the time spent on an incremental fetch goes
      down, as our delta heuristics are able to do their work.
      And we save time on the partial bitmap clone for the same
      reason.
      Signed-off-by: default avatarVicent Marti <tanoku@gmail.com>
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      ae4f07fb
    • Vicent Marti's avatar
      pack-objects: implement bitmap writing · 7cc8f971
      Vicent Marti authored
      This commit extends more the functionality of `pack-objects` by allowing
      it to write out a `.bitmap` index next to any written packs, together
      with the `.idx` index that currently gets written.
      
      If bitmap writing is enabled for a given repository (either by calling
      `pack-objects` with the `--write-bitmap-index` flag or by having
      `pack.writebitmaps` set to `true` in the config) and pack-objects is
      writing a packfile that would normally be indexed (i.e. not piping to
      stdout), we will attempt to write the corresponding bitmap index for the
      packfile.
      
      Bitmap index writing happens after the packfile and its index has been
      successfully written to disk (`finish_tmp_packfile`). The process is
      performed in several steps:
      
          1. `bitmap_writer_set_checksum`: this call stores the partial
             checksum for the packfile being written; the checksum will be
             written in the resulting bitmap index to verify its integrity
      
          2. `bitmap_writer_build_type_index`: this call uses the array of
             `struct object_entry` that has just been sorted when writing out
             the actual packfile index to disk to generate 4 type-index bitmaps
             (one for each object type).
      
             These bitmaps have their nth bit set if the given object is of
             the bitmap's type. E.g. the nth bit of the Commits bitmap will be
             1 if the nth object in the packfile index is a commit.
      
             This is a very cheap operation because the bitmap writing code has
             access to the metadata stored in the `struct object_entry` array,
             and hence the real type for each object in the packfile.
      
          3. `bitmap_writer_reuse_bitmaps`: if there exists an existing bitmap
             index for one of the packfiles we're trying to repack, this call
             will efficiently rebuild the existing bitmaps so they can be
             reused on the new index. All the existing bitmaps will be stored
             in a `reuse` hash table, and the commit selection phase will
             prioritize these when selecting, as they can be written directly
             to the new index without having to perform a revision walk to
             fill the bitmap. This can greatly speed up the repack of a
             repository that already has bitmaps.
      
          4. `bitmap_writer_select_commits`: if bitmap writing is enabled for
             a given `pack-objects` run, the sequence of commits generated
             during the Counting Objects phase will be stored in an array.
      
             We then use that array to build up the list of selected commits.
             Writing a bitmap in the index for each object in the repository
             would be cost-prohibitive, so we use a simple heuristic to pick
             the commits that will be indexed with bitmaps.
      
             The current heuristics are a simplified version of JGit's
             original implementation. We select a higher density of commits
             depending on their age: the 100 most recent commits are always
             selected, after that we pick 1 commit of each 100, and the gap
             increases as the commits grow older. On top of that, we make sure
             that every single branch that has not been merged (all the tips
             that would be required from a clone) gets their own bitmap, and
             when selecting commits between a gap, we tend to prioritize the
             commit with the most parents.
      
             Do note that there is no right/wrong way to perform commit
             selection; different selection algorithms will result in
             different commits being selected, but there's no such thing as
             "missing a commit". The bitmap walker algorithm implemented in
             `prepare_bitmap_walk` is able to adapt to missing bitmaps by
             performing manual walks that complete the bitmap: the ideal
             selection algorithm, however, would select the commits that are
             more likely to be used as roots for a walk in the future (e.g.
             the tips of each branch, and so on) to ensure a bitmap for them
             is always available.
      
          5. `bitmap_writer_build`: this is the computationally expensive part
             of bitmap generation. Based on the list of commits that were
             selected in the previous step, we perform several incremental
             walks to generate the bitmap for each commit.
      
             The walks begin from the oldest commit, and are built up
             incrementally for each branch. E.g. consider this dag where A, B,
             C, D, E, F are the selected commits, and a, b, c, e are a chunk
             of simplified history that will not receive bitmaps.
      
                  A---a---B--b--C--c--D
                           \
                            E--e--F
      
             We start by building the bitmap for A, using A as the root for a
             revision walk and marking all the objects that are reachable
             until the walk is over. Once this bitmap is stored, we reuse the
             bitmap walker to perform the walk for B, assuming that once we
             reach A again, the walk will be terminated because A has already
             been SEEN on the previous walk.
      
             This process is repeated for C, and D, but when we try to
             generate the bitmaps for E, we can reuse neither the current walk
             nor the bitmap we have generated so far.
      
             What we do now is resetting both the walk and clearing the
             bitmap, and performing the walk from scratch using E as the
             origin. This new walk, however, does not need to be completed.
             Once we hit B, we can lookup the bitmap we have already stored
             for that commit and OR it with the existing bitmap we've composed
             so far, allowing us to limit the walk early.
      
             After all the bitmaps have been generated, another iteration
             through the list of commits is performed to find the best XOR
             offsets for compression before writing them to disk. Because of
             the incremental nature of these bitmaps, XORing one of them with
             its predecesor results in a minimal "bitmap delta" most of the
             time. We can write this delta to the on-disk bitmap index, and
             then re-compose the original bitmaps by XORing them again when
             loaded.
      
             This is a phase very similar to pack-object's `find_delta` (using
             bitmaps instead of objects, of course), except the heuristics
             have been greatly simplified: we only check the 10 bitmaps before
             any given one to find best compressing one. This gives good
             results in practice, because there is locality in the ordering of
             the objects (and therefore bitmaps) in the packfile.
      
           6. `bitmap_writer_finish`: the last step in the process is
      	serializing to disk all the bitmap data that has been generated
      	in the two previous steps.
      
      	The bitmap is written to a tmp file and then moved atomically to
      	its final destination, using the same process as
      	`pack-write.c:write_idx_file`.
      Signed-off-by: default avatarVicent Marti <tanoku@gmail.com>
      Signed-off-by: default avatarJeff King <peff@peff.net>
      Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
      7cc8f971