1. 13 Feb, 2019 1 commit
    • Duy Nguyen's avatar
      read-cache.c: fix writing "link" index ext with null base oid · 6e37c8ed
      Duy Nguyen authored
      Since commit 7db11830 (unpack_trees: fix breakage when o->src_index !=
      o->dst_index - 2018-04-23) and changes in merge code to use separate
      index_state for source and destination, when doing a merge with split
      index activated, we may run into this line in unpack_trees():
      
          o->result.split_index = init_split_index(&o->result);
      
      This is by itself not wrong. But this split index information is not
      fully populated (and it's only so when move_cache_to_base_index() is
      called, aka force splitting the index, or loading index_state from a
      file). Both "base_oid" and "base" in this case remain null.
      
      So when writing the main index down, we link to this index with null
      oid (default value after init_split_index()), which also means "no split
      index" internally. This triggers an incorrect base index refresh:
      
          warning: could not freshen shared index '.../sharedindex.0{40}'
      
      This patch makes sure we will not refresh null base_oid (because the
      file is never there). It also makes sure not to write "link" extension
      with null base_oid in the first place (no point having it at
      all). Read code already has protection against null base_oid.
      
      There is also another side fix in remove_split_index() that causes a
      crash when doing "git update-index --no-split-index" when base_oid in
      the index file is null. In this case we will not load
      istate->split_index->base but we dereference it anyway and are rewarded
      with a segfault. This should not happen anymore, but it's still wrong to
      dereference a potential NULL pointer, especially when we do check for
      NULL pointer in the next code.
      Reported-by: Luke Diamand's avatarLuke Diamand <luke@diamand.org>
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      6e37c8ed
  2. 02 Jan, 2019 1 commit
    • Thomas Gummerer's avatar
      read-cache: add invalidate parameter to remove_marked_cache_entries · 6fdc2057
      Thomas Gummerer authored
      When marking cache entries for removal, and later removing them all at
      once using 'remove_marked_cache_entries()', cache entries currently
      have to be invalidated manually in the cache tree and in the untracked
      cache.
      
      Add an invalidate flag to the function.  With the flag set, the
      function will take care of invalidating the path in the cache tree and
      in the untracked cache.
      
      Note that the current callsites already do the invalidation properly
      in other places, so we're just passing 0 from there to keep the status
      quo.
      
      This will be useful in a subsequent commit.
      Signed-off-by: 's avatarThomas Gummerer <t.gummerer@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      6fdc2057
  3. 11 Oct, 2018 4 commits
    • Gábor Szeder's avatar
      split-index: BUG() when cache entry refers to non-existing shared entry · 4c490f3d
      Gábor Szeder authored
      When the split index feature is in use, then a cache entry is:
      
        - either only present in the split index, in which case its 'index'
          field must be 0,
      
        - or it should refer to an existing entry in the shared index, i.e.
          the 'index' field can't be greater than the size of the shared
          index.
      
      If a cache entry were to refer to a non-existing entry in the shared
      index, then that's a sign of something being wrong in the index state,
      either as a result of a bug in dealing with the split/shared index
      entries, or perhaps a (potentially unrelated) memory corruption issue.
      
      prepare_to_write_split_index() already has a condition to catch cache
      entries with such bogus 'index' field, but instead of calling BUG() it
      just sets cache entry's 'index = 0', and the entry will then be
      written to the new split index.
      
      Don't write a new index file from bogus index state, and call BUG()
      upon encountering an cache entry referring to a non-existing shared
      index entry.
      
      Running the test suite repeatedly with 'GIT_TEST_SPLIT_INDEX=yes'
      doesn't trigger this condition.
      Signed-off-by: Gábor Szeder's avatarSZEDER Gábor <szeder.dev@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      4c490f3d
    • Gábor Szeder's avatar
      split-index: smudge and add racily clean cache entries to split index · 5581a019
      Gábor Szeder authored
      Ever since the split index feature was introduced [1], refreshing a
      split index is prone to a variant of the classic racy git problem.
      
      Consider the following sequence of commands updating the split index
      when the shared index contains a racily clean cache entry, i.e. an
      entry whose cached stat data matches with the corresponding file in
      the worktree and the cached mtime matches that of the index:
      
        echo "cached content" >file
        git update-index --split-index --add file
        echo "dirty worktree" >file    # size stays the same!
        # ... wait ...
        git update-index --add other-file
      
      Normally, when a non-split index is updated, then do_write_index()
      (the function responsible for writing all kinds of indexes, "regular",
      split, and shared) recognizes racily clean cache entries, and writes
      them with smudged stat data, i.e. with file size set to 0.  When
      subsequent git commands read the index, they will notice that the
      smudged stat data doesn't match with the file in the worktree, and
      then go on to check the file's content and notice its dirtiness.
      
      In the above example, however, in the second 'git update-index'
      prepare_to_write_split_index() decides which cache entries stored only
      in the shared index should be replaced in the new split index.  Alas,
      this function never looks out for racily clean cache entries, and
      since the file's stat data in the worktree hasn't changed since the
      shared index was written, it won't be replaced in the new split index.
      Consequently, do_write_index() doesn't even get this racily clean
      cache entry, and can't smudge its stat data.  Subsequent git commands
      will then see that the index has more recent mtime than the file and
      that the (not smudged) cached stat data still matches with the file in
      the worktree, and, ultimately, will erroneously consider the file
      clean.
      
      Modify prepare_to_write_split_index() to recognize racily clean cache
      entries, and mark them to be added to the split index.  Note that
      there are two places where it should check raciness: first those cache
      entries that are only stored in the shared index, and then those that
      have been copied by unpack_trees() from the shared index while it
      constructed a new index.  This way do_write_index() will get these
      racily clean cache entries as well, and will then write them with
      smudged stat data to the new split index.
      
      This change makes all tests in 't1701-racy-split-index.sh' pass, so
      flip the two 'test_expect_failure' tests to success.  Also add the '#'
      (as in nr. of trial) to those tests' description that were omitted
      when the tests expected failure.
      
      Note that after this change if the index is split when it contains a
      racily clean cache entry, then a smudged cache entry will be written
      both to the new shared and to the new split indexes.  This doesn't
      affect regular git commands: as far as they are concerned this is just
      an entry in the split index replacing an outdated entry in the shared
      index.  It did affect a few tests in 't1700-split-index.sh', though,
      because they actually check which entries are stored in the split
      index; a previous patch in this series has already made the necessary
      adjustments in 't1700'.  And racily clean cache entries and index
      splitting are rare enough to not worry about the resulting duplicated
      smudged cache entries, and the additional complexity required to
      prevent them is not worth it.
      
      Several tests failed occasionally when the test suite was run with
      'GIT_TEST_SPLIT_INDEX=yes'.  Here are those that I managed to trace
      back to this racy split index problem, starting with those failing
      more frequently, with a link to a failing Travis CI build job for
      each.  The highlighted line [2] shows when the racy file was written,
      which is not always in the failing test but in a preceeding setup
      test.
      
        t3903-stash.sh:
          https://travis-ci.org/git/git/jobs/385542084#L5858
      
        t4024-diff-optimize-common.sh:
          https://travis-ci.org/git/git/jobs/386531969#L3174
      
        t4015-diff-whitespace.sh:
          https://travis-ci.org/git/git/jobs/360797600#L8215
      
        t2200-add-update.sh:
          https://travis-ci.org/git/git/jobs/382543426#L3051
      
        t0090-cache-tree.sh:
          https://travis-ci.org/git/git/jobs/416583010#L3679
      
      There might be others, e.g. perhaps 't1000-read-tree-m-3way.sh' and
      others using 'lib-read-tree-m-3way.sh', but I couldn't confirm yet.
      
      [1] In the branch leading to the merge commit v2.1.0-rc0~45 (Merge
          branch 'nd/split-index', 2014-07-16).
      
      [2] Note that those highlighted lines are in the 'after failure' fold,
          and your browser might unhelpfully fold it up before you could
          take a good look.
      Signed-off-by: Gábor Szeder's avatarSZEDER Gábor <szeder.dev@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      5581a019
    • Gábor Szeder's avatar
      split-index: don't compare cached data of entries already marked for split index · e3d83798
      Gábor Szeder authored
      When unpack_trees() constructs a new index, it copies cache entries
      from the original index [1].  prepare_to_write_split_index() has to
      deal with this, and it has a dedicated code path for copied entries
      that are present in the shared index, where it compares the cached
      data in the corresponding copied and original entries.  If the cached
      data matches, then they are considered the same; if it differs, then
      the copied entry will be marked for inclusion as a replacement entry
      in the just about to be written split index by setting the
      CE_UPDATE_IN_BASE flag.
      
      However, a cache entry already has its CE_UPDATE_IN_BASE flag set upon
      reading the split index, if the entry already has a replacement entry
      there, or upon refreshing the cached stat data, if the corresponding
      file was modified.  The state of this flag is then preserved when
      unpack_trees() copies a cache entry from the shared index.
      
      So modify prepare_to_write_split_index() to check the copied cache
      entries' CE_UPDATE_IN_BASE flag first, and skip the thorough
      comparison of cached data if the flag is already set.  Those couple of
      lines comparing the cached data would then have too many levels of
      indentation, so extract them into a helper function.
      
      Note that comparing the cached data in copied and original entries in
      the shared index might actually be entirely unnecessary.  In theory
      all code paths refreshing the cached stat data of an entry in the
      shared index should set the CE_UPDATE_IN_BASE flag in that entry, and
      unpack_trees() should preserve this flag when copying cache entries.
      This means that the cached data is only ever changed if the
      CE_UPDATE_IN_BASE flag is set as well.  Our test suite seems to
      confirm this: instrumenting the conditions in question and running the
      test suite repeatedly with 'GIT_TEST_SPLIT_INDEX=yes' showed that the
      cached data in a copied entry differs from the data in the shared
      entry only if its CE_UPDATE_IN_BASE flag is indeed set.
      
      In practice, however, our test suite doesn't have 100% coverage,
      GIT_TEST_SPLIT_INDEX is inherently random, and I certainly can't claim
      to possess complete understanding of what goes on in unpack_trees()...
      Therefore I kept the comparison of the cached data when
      CE_UPDATE_IN_BASE is not set, just in case that an unnoticed or future
      code path were to accidentally miss setting this flag upon refreshing
      the cached stat data or unpack_trees() were to drop this flag while
      copying a cache entry.
      
      [1] Note that when unpack_trees() constructs the new index and decides
          that a cache entry should now refer to different content than what
          was recorded in the original index (e.g. 'git read-tree -m
          HEAD^'), then that can't really be considered a copy of the
          original, but rather the creation of a new entry.  Notably and
          pertinent to the split index feature, such a new entry doesn't
          have a reference to the original's shared index entry anymore,
          i.e. its 'index' field is set to 0.  Consequently, such an entry
          is treated by prepare_to_write_split_index() as an entry not
          present in the shared index and it will be added to the new split
          index, while the original entry will be marked as deleted, and
          neither the above discussion nor the changes in this patch apply
          to them.
      Signed-off-by: Gábor Szeder's avatarSZEDER Gábor <szeder.dev@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      e3d83798
    • Gábor Szeder's avatar
      split-index: count the number of deleted entries · 2034e848
      Gábor Szeder authored
      'struct split_index' contains the field 'nr_deletions', whose name
      with the 'nr_' prefix suggests that it contains the number of deleted
      cache entries.  However, barring its initialization to 0, this field
      is only ever set to 1, indicating that there is at least one deleted
      entry, but not the number of deleted entries.  Luckily, this doesn't
      cause any issues (other than confusing the reader, that is), because
      the only place reading this field uses it in the same sense, i.e.: 'if
      (si->nr_deletions)'.
      
      To avoid confusion, we could either rename this field to something
      like 'has_deletions' to make its name match its role, or make it a
      counter of deleted cache entries to match its name.
      
      Let's make it a counter, to keep it in sync with the related field
      'nr_replacements', which does contain the number of replaced cache
      entries.  This will also give developers debugging the split index
      code easy access to the number of deleted cache entries.
      Signed-off-by: Gábor Szeder's avatarSZEDER Gábor <szeder.dev@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      2034e848
  4. 03 Jul, 2018 2 commits
    • Jameson Miller's avatar
      block alloc: allocate cache entries from mem_pool · 8e72d675
      Jameson Miller authored
      When reading large indexes from disk, a portion of the time is
      dominated in malloc() calls. This can be mitigated by allocating a
      large block of memory and manage it ourselves via memory pools.
      
      This change moves the cache entry allocation to be on top of memory
      pools.
      
      Design:
      
      The index_state struct will gain a notion of an associated memory_pool
      from which cache_entries will be allocated from. When reading in the
      index from disk, we have information on the number of entries and
      their size, which can guide us in deciding how large our initial
      memory allocation should be. When an index is discarded, the
      associated memory_pool will be discarded as well - so the lifetime of
      a cache_entry is tied to the lifetime of the index_state that it was
      allocated for.
      
      In the case of a Split Index, the following rules are followed. 1st,
      some terminology is defined:
      
      Terminology:
        - 'the_index': represents the logical view of the index
      
        - 'split_index': represents the "base" cache entries. Read from the
          split index file.
      
      'the_index' can reference a single split_index, as well as
      cache_entries from the split_index. `the_index` will be discarded
      before the `split_index` is.  This means that when we are allocating
      cache_entries in the presence of a split index, we need to allocate
      the entries from the `split_index`'s memory pool.  This allows us to
      follow the pattern that `the_index` can reference cache_entries from
      the `split_index`, and that the cache_entries will not be freed while
      they are still being referenced.
      
      Managing transient cache_entry structs:
      Cache entries are usually allocated for an index, but this is not always
      the case. Cache entries are sometimes allocated because this is the
      type that the existing checkout_entry function works with. Because of
      this, the existing code needs to handle cache entries associated with an
      index / memory pool, and those that only exist transiently. Several
      strategies were contemplated around how to handle this:
      
      Chosen approach:
      An extra field was added to the cache_entry type to track whether the
      cache_entry was allocated from a memory pool or not. This is currently
      an int field, as there are no more available bits in the existing
      ce_flags bit field. If / when more bits are needed, this new field can
      be turned into a proper bit field.
      
      Alternatives:
      
      1) Do not include any information about how the cache_entry was
      allocated. Calling code would be responsible for tracking whether the
      cache_entry needed to be freed or not.
        Pro: No extra memory overhead to track this state
        Con: Extra complexity in callers to handle this correctly.
      
      The extra complexity and burden to not regress this behavior in the
      future was more than we wanted.
      
      2) cache_entry would gain knowledge about which mem_pool allocated it
        Pro: Could (potentially) do extra logic to know when a mem_pool no
             longer had references to any cache_entry
        Con: cache_entry would grow heavier by a pointer, instead of int
      
      We didn't see a tangible benefit to this approach
      
      3) Do not add any extra information to a cache_entry, but when freeing a
         cache entry, check if the memory exists in a region managed by existing
         mem_pools.
        Pro: No extra memory overhead to track state
        Con: Extra computation is performed when freeing cache entries
      
      We decided tracking and iterating over known memory pool regions was
      less desirable than adding an extra field to track this stae.
      Signed-off-by: 's avatarJameson Miller <jamill@microsoft.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      8e72d675
    • Jameson Miller's avatar
      block alloc: add lifecycle APIs for cache_entry structs · a849735b
      Jameson Miller authored
      It has been observed that the time spent loading an index with a large
      number of entries is partly dominated by malloc() calls. This change
      is in preparation for using memory pools to reduce the number of
      malloc() calls made to allocate cahce entries when loading an index.
      
      Add an API to allocate and discard cache entries, abstracting the
      details of managing the memory backing the cache entries. This commit
      does actually change how memory is managed - this will be done in a
      later commit in the series.
      
      This change makes the distinction between cache entries that are
      associated with an index and cache entries that are not associated with
      an index. A main use of cache entries is with an index, and we can
      optimize the memory management around this. We still have other cases
      where a cache entry is not persisted with an index, and so we need to
      handle the "transient" use case as well.
      
      To keep the congnitive overhead of managing the cache entries, there
      will only be a single discard function. This means there must be enough
      information kept with the cache entry so that we know how to discard
      them.
      
      A summary of the main functions in the API is:
      
      make_cache_entry: create cache entry for use in an index. Uses specified
                        parameters to populate cache_entry fields.
      
      make_empty_cache_entry: Create an empty cache entry for use in an index.
                              Returns cache entry with empty fields.
      
      make_transient_cache_entry: create cache entry that is not used in an
                                  index. Uses specified parameters to populate
                                  cache_entry fields.
      
      make_empty_transient_cache_entry: create cache entry that is not used in
                                        an index. Returns cache entry with
                                        empty fields.
      
      discard_cache_entry: A single function that knows how to discard a cache
                           entry regardless of how it was allocated.
      Signed-off-by: 's avatarJameson Miller <jamill@microsoft.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      a849735b
  5. 02 May, 2018 1 commit
  6. 22 Feb, 2018 1 commit
  7. 19 Jan, 2018 1 commit
    • Thomas Gummerer's avatar
      split-index: don't write cache tree with null oid entries · 4bddd983
      Thomas Gummerer authored
      In a96d3cc3 ("cache-tree: reject entries with null sha1", 2017-04-21)
      we made sure that broken cache entries do not get propagated to new
      trees.  Part of that was making sure not to re-use an existing cache
      tree that includes a null oid.
      
      It did so by dropping the cache tree in 'do_write_index()' if one of
      the entries contains a null oid.  In split index mode however, there
      are two invocations to 'do_write_index()', one for the shared index
      and one for the split index.  The cache tree is only written once, to
      the split index.
      
      As we only loop through the elements that are effectively being
      written by the current invocation, that may not include the entry with
      a null oid in the split index (when it is already written to the
      shared index), where we write the cache tree.  Therefore in split
      index mode we may still end up writing the cache tree, even though
      there is an entry with a null oid in the index.
      
      Fix this by checking for null oids in prepare_to_write_split_index,
      where we loop the entries of the shared index as well as the entries for
      the split index.
      
      This fixes t7009 with GIT_TEST_SPLIT_INDEX.  Also add a new test that's
      more specifically showing the problem.
      Signed-off-by: 's avatarThomas Gummerer <t.gummerer@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      4bddd983
  8. 24 Jun, 2017 1 commit
  9. 16 Jun, 2017 1 commit
  10. 08 May, 2017 1 commit
    • Duy Nguyen's avatar
      split-index: add and use unshare_split_index() · f9d7abec
      Duy Nguyen authored
      When split-index is being used, we have two cache_entry arrays in
      index_state->cache[] and index_state->split_index->base->cache[].
      
      index_state->cache[] may share the same entries with base->cache[] so
      we can quickly determine what entries are shared. This makes memory
      management tricky, we can't free base->cache[] until we know
      index_state->cache[] does not point to any of those entries.
      
      unshare_split_index() is added for this purpose, to find shared
      entries and either duplicate them in index_state->cache[], or discard
      them. Either way it should be safe to free base->cache[] after
      unshare_split_index().
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      f9d7abec
  11. 01 Mar, 2017 1 commit
  12. 01 Nov, 2016 1 commit
  13. 25 Sep, 2016 1 commit
    • René Scharfe's avatar
      use COPY_ARRAY · 45ccef87
      René Scharfe authored
      Add a semantic patch for converting certain calls of memcpy(3) to
      COPY_ARRAY() and apply that transformation to the code base.  The result
      is
       shorter and safer code.  For now only consider calls where source and
      destination have the same type, or in other words: easy cases.
      Signed-off-by: 's avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      45ccef87
  14. 06 May, 2016 1 commit
    • Li Peng's avatar
      typofix: assorted typofixes in comments, documentation and messages · 832c0e5e
      Li Peng authored
      Many instances of duplicate words (e.g. "the the path") and
      a few typoes are fixed, originally in multiple patches.
      
          wildmatch: fix duplicate words of "the"
          t: fix duplicate words of "output"
          transport-helper: fix duplicate words of "read"
          Git.pm: fix duplicate words of "return"
          path: fix duplicate words of "look"
          pack-protocol.txt: fix duplicate words of "the"
          precompose-utf8: fix typo of "sequences"
          split-index: fix typo
          worktree.c: fix typo
          remote-ext: fix typo
          utf8: fix duplicate words of "the"
          git-cvsserver: fix duplicate words
      Signed-off-by: 's avatarLi Peng <lip@dtdream.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      832c0e5e
  15. 12 Mar, 2015 1 commit
  16. 13 Jun, 2014 8 commits
    • Duy Nguyen's avatar
      update-index: new options to enable/disable split index mode · c18b80a0
      Duy Nguyen authored
      If you have a large work tree but only make changes in a subset, then
      $GIT_DIR/index's size should be stable after a while. If you change
      branches that touch something else, $GIT_DIR/index's size may grow
      large that it becomes as slow as the unified index. Do --split-index
      again occasionally to force all changes back to the shared index and
      keep $GIT_DIR/index small.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      c18b80a0
    • Duy Nguyen's avatar
      split-index: strip pathname of on-disk replaced entries · b3c96fb1
      Duy Nguyen authored
      We know the positions of replaced entries via the replace bitmap in
      "link" extension, so the "name" path does not have to be stored (it's
      still in the shared index). With this, we also have a way to
      distinguish additions vs replacements at load time and can catch
      broken "link" extensions.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      b3c96fb1
    • Duy Nguyen's avatar
      split-index: do not invalidate cache-tree at read time · ce7c614b
      Duy Nguyen authored
      We are sure that after merge_base_index() is done. cache-tree can
      still be used with the final index. So don't destroy cache tree.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      ce7c614b
    • Duy Nguyen's avatar
      split-index: the reading part · 76b07b37
      Duy Nguyen authored
      CE_REMOVE'd entries are removed here because only parts of the code
      base (unpack_trees in fact) test this bit when they look for the
      presence of an entry. Leaving them may confuse the code ignores this
      bit and expects to see a real entry.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      76b07b37
    • Duy Nguyen's avatar
      split-index: the writing part · 96a1d8d3
      Duy Nguyen authored
      prepare_to_write_split_index() does the major work, classifying
      deleted, updated and added entries. write_link_extension() then just
      writes it down.
      
      An observation is, deleting an entry, then adding it back is recorded
      as "entry X is deleted, entry X is added", not "entry X is replaced".
      This is simpler, with small overhead: a replaced entry is stored
      without its path, a new entry is store with its path.
      
      A note about unpack_trees() and the deduplication code inside
      prepare_to_write_split_index(). Usually tracking updated/removed
      entries via read-cache API is enough. unpack_trees() manipulates the
      index in a different way: it throws the entire source index out,
      builds up a new one, copying/duplicating entries (using dup_entry)
      from the source index over if necessary, then returns the new index.
      
      A naive solution would be marking the entire source index "deleted"
      and add their duplicates as new. That could bring $GIT_DIR/index back
      to the original size. So we try harder and memcmp() between the
      original and the duplicate to see if it needs updating.
      
      We could avoid memcmp() too, by avoiding duplicating the original
      entry in dup_entry(). The performance gain this way is within noise
      level and it complicates unpack-trees.c. So memcmp() is the preferred
      way to deal with deduplication.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      96a1d8d3
    • Duy Nguyen's avatar
      read-cache: mark updated entries for split index · 078a58e8
      Duy Nguyen authored
      The large part of this patch just follows CE_ENTRY_CHANGED
      marks. replace_index_entry() is updated to update
      split_index->base->cache[] as well so base->cache[] does not reference
      to a freed entry.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      078a58e8
    • Duy Nguyen's avatar
      read-cache: save deleted entries in split index · 045113a5
      Duy Nguyen authored
      Entries that belong to the base index should not be freed. Mark
      CE_REMOVE to track them.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      045113a5
    • Duy Nguyen's avatar
      read-cache: split-index mode · 5fc2fc8f
      Duy Nguyen authored
      This split-index mode is designed to keep write cost proportional to
      the number of changes the user has made, not the size of the work
      tree. (Read cost is another matter, to be dealt separately.)
      
      This mode stores index info in a pair of $GIT_DIR/index and
      $GIT_DIR/sharedindex.<SHA-1>. sharedindex is large and unchanged over
      time while "index" is smaller and updated often. Format details are in
      index-format.txt, although not everything is implemented in this
      patch.
      
      Shared indexes are not automatically removed, because it's unclear if
      the shared index is needed by any (even temporary) indexes by just
      looking at it. After a while you'll collect stale shared indexes. The
      good news is one shared index is useable for long, until
      $GIT_DIR/index becomes too big and sluggish that the new shared index
      must be created.
      
      The safest way to clean shared indexes is to turn off split index
      mode, so shared files are all garbage, delete them all, then turn on
      split index mode again.
      Signed-off-by: Duy Nguyen's avatarNguyễn Thái Ngọc Duy <pclouds@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      5fc2fc8f