1. 01 Apr, 2019 2 commits
  2. 12 Nov, 2018 1 commit
  3. 04 Sep, 2018 1 commit
    • Jeff King's avatar
      bitmap_has_sha1_in_uninteresting(): drop BUG check · 5476fb07
      Jeff King authored
      Commit 30cdc33f (pack-bitmap: save "have" bitmap from
      walk, 2018-08-21) introduced a new function for looking at
      the "have" side of a bitmap walk. Because it only makes
      sense to do so after we've finished the walk, we added an
      extra safety assertion, making sure that bitmap_git->result
      is non-NULL.
      However, this safety is misguided. It was trying to catch
      the case where we had called prepare_bitmap_walk() to give
      us a "struct bitmap_index", but had not yet called
      traverse_bitmap_commit_list() to walk it. But all of the
      interesting computation (including setting up the result and
      "have" bitmaps) happens in the first function! The latter
      function only delivers the result to a callback function.
      So the case we were worried about is impossible; if you get
      a non-NULL result from prepare_bitmap_walk(), then its
      "have" field will be fully formed.
      But much worse, traverse_bitmap_commit_list() actually frees
      the result field as it finishes. Which means that this
      assertion is worse than useless: it's almost guaranteed to
      Our test suite didn't catch this because the function isn't
      actually exercised at all. The only caller comes from
      6a1e32d5 (pack-objects: reuse on-disk deltas for thin
      "have" objects, 2018-08-21), and that's triggered only when
      you fetch or push history that contains an object with a
      base that is found deep in history. Our test suite fetches
      and pushes either don't use bitmaps, or use too-small
      example repositories. But any reasonably-sized real-world
      push or fetch (with bitmaps) would trigger this.
      This patch drops the harmful assertion and tweaks the
      docstring for the function to make the precondition clear.
      The tests need to be improved to exercise this new
      pack-objects feature, but we'll do that in a separate
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
  4. 21 Aug, 2018 1 commit
    • Jeff King's avatar
      pack-bitmap: save "have" bitmap from walk · 30cdc33f
      Jeff King authored
      When we do a bitmap walk, we save the result, which
      represents (WANTs & ~HAVEs); i.e., every object we care
      about visiting in our walk. However, we throw away the
      haves bitmap, which can sometimes be useful, too. Save it
      and provide an access function so code which has performed a
      walk can query it.
      A few notes on the accessor interface:
       - the bitmap code calls these "haves" because it grew out
         of the want/have negotiation for fetches. But really,
         these are simply the objects that would be flagged
         UNINTERESTING in a regular traversal. Let's use that
         more universal nomenclature for the external module
         interface. We may want to change the internal naming
         inside the bitmap code, but that's outside the scope of
         this patch.
       - it still uses a bare "sha1" rather than "oid". That's
         true of all of the bitmap code. And in this particular
         instance, our caller in pack-objects is dealing with the
         bare sha1 that comes from a packed REF_DELTA (we're
         pointing directly to the mmap'd pack on disk). That's
         something we'll have to deal with as we transition to a
         new hash, but we can wait and see how the caller ends up
         being fixed and adjust this interface accordingly.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
  5. 15 Aug, 2018 1 commit
  6. 21 Jun, 2018 2 commits
    • Jonathan Tan's avatar
      pack-bitmap: add free function · f3c23db2
      Jonathan Tan authored
      Add a function to free struct bitmap_index instances, and use it where
      needed (except when rebuild_existing_bitmaps() is used, since it creates
      references to the bitmaps within the struct bitmap_index passed to it).
      Note that the hashes field in struct bitmap_index is not freed because
      it points to another field within the same struct. The documentation for
      that field has been updated to clarify that.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
    • Jonathan Tan's avatar
      pack-bitmap: remove bitmap_git global variable · 3ae5fa07
      Jonathan Tan authored
      Remove the bitmap_git global variable. Instead, generate on demand an
      instance of struct bitmap_index for code that needs to access it.
      This allows us significant control over the lifetime of instances of
      struct bitmap_index. In particular, packs can now be closed without
      worrying if an unnecessarily long-lived "pack" field in struct
      bitmap_index still points to it.
      The bitmap API is also clearer in that we need to first obtain a struct
      bitmap_index, then we use it.
      This patch raises two potential issues: (1) memory for the struct
      bitmap_index is allocated without being freed, and (2)
      prepare_bitmap_git() and prepare_bitmap_walk() can reuse a previously
      loaded bitmap. For (1), this will be dealt with in a subsequent patch in
      this patch set that also deals with freeing the contents of the struct
      bitmap_index (which were not freed previously, because they have global
      scope). For (2), current bitmap users only load the bitmap once at most
      (note that pack-objects can use bitmaps or write bitmaps, but not both
      at the same time), so support for reuse has no effect - and future users
      can pass around the struct bitmap_index * obtained if they need to do 2
      or more things with the same bitmap.
      Helped-by: Stefan Beller's avatarStefan Beller <sbeller@google.com>
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Helped-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
  7. 16 Apr, 2018 1 commit
  8. 16 Oct, 2017 1 commit
  9. 15 Jan, 2015 1 commit
  10. 01 Dec, 2014 1 commit
    • Karsten Blees's avatar
      pack-bitmap: do not use gcc packed attribute · b5007211
      Karsten Blees authored
      The "__attribute__" flag may be a noop on some compilers.
      That's OK as long as the code is correct without the
      attribute, but in this case it is not. We would typically
      end up with a struct that is 2 bytes too long due to struct
      padding, breaking both reading and writing of bitmaps.
      Instead of marshalling the data in a struct, let's just
      provide helpers for reading and writing the appropriate
      types. Besides being correct on all platforms, the result is
      more efficient and simpler to read.
      Signed-off-by: 's avatarKarsten Blees <blees@dcon.de>
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
  11. 30 Dec, 2013 3 commits
    • Vicent Marti's avatar
      pack-bitmap: implement optional name_hash cache · ae4f07fb
      Vicent Marti authored
      When we use pack bitmaps rather than walking the object
      graph, we end up with the list of objects to include in the
      packfile, but we do not know the path at which any tree or
      blob objects would be found.
      In a recently packed repository, this is fine. A fetch would
      use the paths only as a heuristic in the delta compression
      phase, and a fully packed repository should not need to do
      much delta compression.
      As time passes, though, we may acquire more objects on top
      of our large bitmapped pack. If clients fetch frequently,
      then they never even look at the bitmapped history, and all
      works as usual. However, a client who has not fetched since
      the last bitmap repack will have "have" tips in the
      bitmapped history, but "want" newer objects.
      The bitmaps themselves degrade gracefully in this
      circumstance. We manually walk the more recent bits of
      history, and then use bitmaps when we hit them.
      But we would also like to perform delta compression between
      the newer objects and the bitmapped objects (both to delta
      against what we know the user already has, but also between
      "new" and "old" objects that the user is fetching). The lack
      of pathnames makes our delta heuristics much less effective.
      This patch adds an optional cache of the 32-bit name_hash
      values to the end of the bitmap file. If present, a reader
      can use it to match bitmapped and non-bitmapped names during
      delta compression.
      Here are perf results for p5310:
      Test                      origin/master       HEAD^                      HEAD
      5310.2: repack to disk    36.81(37.82+1.43)   47.70(48.74+1.41) +29.6%   47.75(48.70+1.51) +29.7%
      5310.3: simulated clone   30.78(29.70+2.14)   1.08(0.97+0.10) -96.5%     1.07(0.94+0.12) -96.5%
      5310.4: simulated fetch   3.16(6.10+0.08)     3.54(10.65+0.06) +12.0%    1.70(3.07+0.06) -46.2%
      5310.6: partial bitmap    36.76(43.19+1.81)   6.71(11.25+0.76) -81.7%    4.08(6.26+0.46) -88.9%
      You can see that the time spent on an incremental fetch goes
      down, as our delta heuristics are able to do their work.
      And we save time on the partial bitmap clone for the same
      Signed-off-by: 's avatarVicent Marti <tanoku@gmail.com>
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
    • Vicent Marti's avatar
      pack-objects: implement bitmap writing · 7cc8f971
      Vicent Marti authored
      This commit extends more the functionality of `pack-objects` by allowing
      it to write out a `.bitmap` index next to any written packs, together
      with the `.idx` index that currently gets written.
      If bitmap writing is enabled for a given repository (either by calling
      `pack-objects` with the `--write-bitmap-index` flag or by having
      `pack.writebitmaps` set to `true` in the config) and pack-objects is
      writing a packfile that would normally be indexed (i.e. not piping to
      stdout), we will attempt to write the corresponding bitmap index for the
      Bitmap index writing happens after the packfile and its index has been
      successfully written to disk (`finish_tmp_packfile`). The process is
      performed in several steps:
          1. `bitmap_writer_set_checksum`: this call stores the partial
             checksum for the packfile being written; the checksum will be
             written in the resulting bitmap index to verify its integrity
          2. `bitmap_writer_build_type_index`: this call uses the array of
             `struct object_entry` that has just been sorted when writing out
             the actual packfile index to disk to generate 4 type-index bitmaps
             (one for each object type).
             These bitmaps have their nth bit set if the given object is of
             the bitmap's type. E.g. the nth bit of the Commits bitmap will be
             1 if the nth object in the packfile index is a commit.
             This is a very cheap operation because the bitmap writing code has
             access to the metadata stored in the `struct object_entry` array,
             and hence the real type for each object in the packfile.
          3. `bitmap_writer_reuse_bitmaps`: if there exists an existing bitmap
             index for one of the packfiles we're trying to repack, this call
             will efficiently rebuild the existing bitmaps so they can be
             reused on the new index. All the existing bitmaps will be stored
             in a `reuse` hash table, and the commit selection phase will
             prioritize these when selecting, as they can be written directly
             to the new index without having to perform a revision walk to
             fill the bitmap. This can greatly speed up the repack of a
             repository that already has bitmaps.
          4. `bitmap_writer_select_commits`: if bitmap writing is enabled for
             a given `pack-objects` run, the sequence of commits generated
             during the Counting Objects phase will be stored in an array.
             We then use that array to build up the list of selected commits.
             Writing a bitmap in the index for each object in the repository
             would be cost-prohibitive, so we use a simple heuristic to pick
             the commits that will be indexed with bitmaps.
             The current heuristics are a simplified version of JGit's
             original implementation. We select a higher density of commits
             depending on their age: the 100 most recent commits are always
             selected, after that we pick 1 commit of each 100, and the gap
             increases as the commits grow older. On top of that, we make sure
             that every single branch that has not been merged (all the tips
             that would be required from a clone) gets their own bitmap, and
             when selecting commits between a gap, we tend to prioritize the
             commit with the most parents.
             Do note that there is no right/wrong way to perform commit
             selection; different selection algorithms will result in
             different commits being selected, but there's no such thing as
             "missing a commit". The bitmap walker algorithm implemented in
             `prepare_bitmap_walk` is able to adapt to missing bitmaps by
             performing manual walks that complete the bitmap: the ideal
             selection algorithm, however, would select the commits that are
             more likely to be used as roots for a walk in the future (e.g.
             the tips of each branch, and so on) to ensure a bitmap for them
             is always available.
          5. `bitmap_writer_build`: this is the computationally expensive part
             of bitmap generation. Based on the list of commits that were
             selected in the previous step, we perform several incremental
             walks to generate the bitmap for each commit.
             The walks begin from the oldest commit, and are built up
             incrementally for each branch. E.g. consider this dag where A, B,
             C, D, E, F are the selected commits, and a, b, c, e are a chunk
             of simplified history that will not receive bitmaps.
             We start by building the bitmap for A, using A as the root for a
             revision walk and marking all the objects that are reachable
             until the walk is over. Once this bitmap is stored, we reuse the
             bitmap walker to perform the walk for B, assuming that once we
             reach A again, the walk will be terminated because A has already
             been SEEN on the previous walk.
             This process is repeated for C, and D, but when we try to
             generate the bitmaps for E, we can reuse neither the current walk
             nor the bitmap we have generated so far.
             What we do now is resetting both the walk and clearing the
             bitmap, and performing the walk from scratch using E as the
             origin. This new walk, however, does not need to be completed.
             Once we hit B, we can lookup the bitmap we have already stored
             for that commit and OR it with the existing bitmap we've composed
             so far, allowing us to limit the walk early.
             After all the bitmaps have been generated, another iteration
             through the list of commits is performed to find the best XOR
             offsets for compression before writing them to disk. Because of
             the incremental nature of these bitmaps, XORing one of them with
             its predecesor results in a minimal "bitmap delta" most of the
             time. We can write this delta to the on-disk bitmap index, and
             then re-compose the original bitmaps by XORing them again when
             This is a phase very similar to pack-object's `find_delta` (using
             bitmaps instead of objects, of course), except the heuristics
             have been greatly simplified: we only check the 10 bitmaps before
             any given one to find best compressing one. This gives good
             results in practice, because there is locality in the ordering of
             the objects (and therefore bitmaps) in the packfile.
           6. `bitmap_writer_finish`: the last step in the process is
      	serializing to disk all the bitmap data that has been generated
      	in the two previous steps.
      	The bitmap is written to a tmp file and then moved atomically to
      	its final destination, using the same process as
      Signed-off-by: 's avatarVicent Marti <tanoku@gmail.com>
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
    • Vicent Marti's avatar
      pack-bitmap: add support for bitmap indexes · fff42755
      Vicent Marti authored
      A bitmap index is a `.bitmap` file that can be found inside
      `$GIT_DIR/objects/pack/`, next to its corresponding packfile, and
      contains precalculated reachability information for selected commits.
      The full specification of the format for these bitmap indexes can be found
      in `Documentation/technical/bitmap-format.txt`.
      For a given commit SHA1, if it happens to be available in the bitmap
      index, its bitmap will represent every single object that is reachable
      from the commit itself. The nth bit in the bitmap is the nth object in
      the packfile; if it's set to 1, the object is reachable.
      By using the bitmaps available in the index, this commit implements
      several new functions:
      	- `prepare_bitmap_git`
      	- `prepare_bitmap_walk`
      	- `traverse_bitmap_commit_list`
      	- `reuse_partial_packfile_from_bitmap`
      The `prepare_bitmap_walk` function tries to build a bitmap of all the
      objects that can be reached from the commit roots of a given `rev_info`
      struct by using the following algorithm:
      - If all the interesting commits for a revision walk are available in
      the index, the resulting reachability bitmap is the bitwise OR of all
      the individual bitmaps.
      - When the full set of WANTs is not available in the index, we perform a
      partial revision walk using the commits that don't have bitmaps as
      roots, and limiting the revision walk as soon as we reach a commit that
      has a corresponding bitmap. The earlier OR'ed bitmap with all the
      indexed commits can now be completed as this walk progresses, so the end
      result is the full reachability list.
      - For revision walks with a HAVEs set (a set of commits that are deemed
      uninteresting), first we perform the same method as for the WANTs, but
      using our HAVEs as roots, in order to obtain a full reachability bitmap
      of all the uninteresting commits. This bitmap then can be used to:
      	a) limit the subsequent walk when building the WANTs bitmap
      	b) finding the final set of interesting commits by performing an
      	   AND-NOT of the WANTs and the HAVEs.
      If `prepare_bitmap_walk` runs successfully, the resulting bitmap is
      stored and the equivalent of a `traverse_commit_list` call can be
      performed by using `traverse_bitmap_commit_list`; the bitmap version
      of this call yields the objects straight from the packfile index
      (without having to look them up or parse them) and hence is several
      orders of magnitude faster.
      As an extra optimization, when `prepare_bitmap_walk` succeeds, the
      `reuse_partial_packfile_from_bitmap` call can be attempted: it will find
      the amount of objects at the beginning of the on-disk packfile that can
      be reused as-is, and return an offset into the packfile. The source
      packfile can then be loaded and the bytes up to `offset` can be written
      directly to the result without having to consider the entires inside the
      packfile individually.
      If the `prepare_bitmap_walk` call fails (e.g. because no bitmap files
      are available), the `rev_info` struct is left untouched, and can be used
      to perform a manual rev-walk using `traverse_commit_list`.
      Hence, this new set of functions are a generic API that allows to
      perform the equivalent of
      	git rev-list --objects [roots...] [^uninteresting...]
      for any set of commits, even if they don't have specific bitmaps
      generated for them.
      In further patches, we'll use this bitmap traversal optimization to
      speed up the `pack-objects` and `rev-list` commands.
      Signed-off-by: 's avatarVicent Marti <tanoku@gmail.com>
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>