1. 15 Apr, 2019 2 commits
    • Jeff King's avatar
      fetch: do not consider peeled tags as advertised tips · 34066f06
      Jeff King authored
      Our filter_refs() function accidentally considers the target of a peeled
      tag to be advertised by the server, even though upload-pack on the
      server side does not consider it so. This can result in the client
      making a bogus fetch to the server, which will end with the server
      complaining "not our ref". Whereas the correct behavior is for the
      client to notice that the server will not allow the request and error
      out immediately.
      
      So as bugs go, this is not very serious (the outcome is the same either
      way -- the fetch fails). But it's worth making the logic here correct
      and consistent with other related cases (e.g., fetching an oid that the
      server did not mention at all).
      
      The crux of the issue comes from fdb69d33 (fetch-pack: always allow
      fetching of literal SHA1s, 2017-05-15). After that, the strategy of
      filter_refs() is basically:
      
        - for each advertised ref, try to match it with a "sought" ref
          provided by the user. Skip any malformed refs (which includes
          peeled values like "refs/tags/foo^{}"), and place any unmatched
          items onto the unmatched list.
      
        - if there are unmatched sought refs, then put all of the advertised
          tips into an oidset, including the unmatched ones.
      
        - for each sought ref, see if it's in the oidset, in which case it's
          legal for us to ask the server for it
      
      The problem is in the second step. Our list of unmatched refs includes
      the peeled refs, even though upload-pack does not allow them to be
      directly fetched. So the simplest fix would be to exclude them during
      that step.
      
      However, we can observe that the unmatched list isn't used for anything
      else, and is freed at the end. We can just free those malformed refs
      immediately. That saves us having to check each ref a second time to see
      if it's malformed.
      
      Note that this code only kicks in when "strict" is in effect. I.e., if
      we are using the v0 protocol and uploadpack.allowReachableSHA1InWant is
      not in effect. With v2, all oids are allowed, and we do not bother
      creating or consulting the oidset at all. To future-proof our test
      against the upcoming GIT_TEST_PROTOCOL_VERSION flag, we'll manually mark
      it as a v0-only test.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      34066f06
    • Jeff King's avatar
      fetch: use free_refs() · 259eddde
      Jeff King authored
      There's no need for us to write this loop manually when a helper
      function can already do it.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      259eddde
  2. 01 Apr, 2019 3 commits
    • Jonathan Tan's avatar
      fetch-pack: binary search when storing wanted-refs · b7643009
      Jonathan Tan authored
      In do_fetch_pack_v2(), the "sought" array is sorted by name, and it is
      not subsequently reordered (within the function). Therefore,
      receive_wanted_refs() can assume that "sought" is sorted, and can thus
      use a binary search when storing wanted-refs retrieved from the server.
      
      Replace the existing linear search with a binary search. This improves
      performance significantly when mirror cloning a repository with more
      than 1 million refs.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      b7643009
    • Jonathan Tan's avatar
      fetch-pack: respect --no-update-shallow in v2 · 1339078f
      Jonathan Tan authored
      In protocol v0, when sending "shallow" lines, the server distinguishes
      between lines caused by the remote repo being shallow and lines caused
      by client-specified depth settings. Unless "--update-shallow" is
      specified, there is a difference in behavior: refs that reach the former
      "shallow" lines, but not the latter, are rejected. But in v2, the server
      does not, and the client treats all "shallow" lines like lines caused by
      client-specified depth settings.
      
      Full restoration of v0 functionality is not possible without protocol
      change, but we can implement a heuristic: if we specify any depth
      setting, treat all "shallow" lines like lines caused by client-specified
      depth settings (that is, unaffected by "--no-update-shallow"), but
      otherwise, treat them like lines caused by the remote repo being shallow
      (that is, affected by "--no-update-shallow"). This restores most of v0
      behavior, except in the case where a client fetches from a shallow
      repository with depth settings.
      
      This patch causes a test that previously failed with
      GIT_TEST_PROTOCOL_VERSION=2 to pass.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      1339078f
    • Jonathan Tan's avatar
      fetch-pack: call prepare_shallow_info only if v0 · 1e7d440b
      Jonathan Tan authored
      In fetch_pack(), be clearer that there is no shallow information before
      the fetch when v2 is used - memset the struct shallow_info to 0 instead
      of calling prepare_shallow_info().
      
      This patch is in preparation for a future patch in which a v2 fetch
      might call prepare_shallow_info() after shallow info has been retrieved
      during the fetch, so I needed to ensure that prepare_shallow_info() is
      not called before the fetch.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      1e7d440b
  3. 20 Mar, 2019 1 commit
    • Jeff King's avatar
      fetch_pack(): drop unused parameters · 0f804b0b
      Jeff King authored
      We don't need the caller of fetch_pack() to pass in "dest", which is the
      remote URL. Since ba227857 (Reduce the number of connects when
      fetching, 2008-02-04), the caller is responsible for calling
      git_connect() itself, and our "dest" parameter is unused.
      
      That commit also started passing us the resulting "conn" child_process
      from git_connect(). But likewise, we do not need do anything with it.
      The descriptors in "fd" are enough for us, and the caller is responsible
      for cleaning up "conn".
      
      We can just drop both parameters.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      0f804b0b
  4. 05 Mar, 2019 1 commit
    • Jeff King's avatar
      fetch: avoid calling write_or_die() · 37c80012
      Jeff King authored
      The write_or_die() function has one quirk that a caller might not
      expect: when it sees EPIPE from the write() call, it translates that
      into a death by SIGPIPE. This doesn't change the overall behavior (the
      program exits either way), but it does potentially confuse test scripts
      looking for a non-signal exit code.
      
      Let's switch away from using write_or_die() in a few code paths, which
      will give us more consistent exit codes. It also gives us the
      opportunity to write more descriptive error messages, since we have
      context that write_or_die() does not.
      
      Note that this won't do much by itself, since we'd typically be killed
      by SIGPIPE before write_or_die() even gets a chance to do its thing.
      That will be addressed in the next patch.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      37c80012
  5. 07 Feb, 2019 1 commit
  6. 04 Feb, 2019 1 commit
    • brian m. carlson's avatar
      fetch-pack: clear alternate shallow when complete · 23311f35
      brian m. carlson authored
      When we write an alternate shallow file in update_shallow, we write it
      into the lock file. The string stored in alternate_shallow_file is
      copied from the lock file path, but it is freed the moment that the lock
      file is closed, since we call strbuf_release to free that path.
      
      This used to work, since we did not invoke git index-pack more than
      once, but now that we do, we reuse the freed memory. Ensure we reset the
      value to NULL to avoid using freed memory. git index-pack will read the
      repository's shallow file, which will have been updated with the correct
      information.
      Signed-off-by: brian m. carlson's avatarbrian m. carlson <sandals@crustytoothpaste.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      23311f35
  7. 17 Jan, 2019 2 commits
    • Jonathan Tan's avatar
      tests: define GIT_TEST_SIDEBAND_ALL · 07c3c2aa
      Jonathan Tan authored
      Define a GIT_TEST_SIDEBAND_ALL environment variable meant to be used
      from tests. When set to true, this overrides uploadpack.allowsidebandall
      to true, allowing the entire test suite to be run as if this
      configuration is in place for all repositories.
      
      As of this patch, all tests pass whether GIT_TEST_SIDEBAND_ALL is unset
      or set to 1.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      07c3c2aa
    • Jonathan Tan's avatar
      {fetch,upload}-pack: sideband v2 fetch response · 0bbc0bc5
      Jonathan Tan authored
      Currently, a response to a fetch request has sideband support only while
      the packfile is being sent, meaning that the server cannot send notices
      until the start of the packfile.
      
      Extend sideband support in protocol v2 fetch responses to the whole
      response. upload-pack will advertise it if the
      uploadpack.allowsidebandall configuration variable is set, and
      fetch-pack will automatically request it if advertised.
      
      If the sideband is to be used throughout the whole response, upload-pack
      will use it to send errors instead of prefixing a PKT-LINE payload with
      "ERR ".
      
      This will be tested in a subsequent patch.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      0bbc0bc5
  8. 15 Jan, 2019 1 commit
  9. 10 Jan, 2019 2 commits
    • Jonathan Tan's avatar
      upload-pack: teach deepen-relative in protocol v2 · 5056cf4a
      Jonathan Tan authored
      Commit 685fbd32 ("fetch-pack: perform a fetch using v2", 2018-03-15)
      attempted to teach Git deepen-relative in protocol v2 (among other
      things), but it didn't work:
      
       (1) fetch-pack.c needs to emit "deepen-relative".
       (2) upload-pack.c needs to ensure that the correct deepen_relative
           variable is passed to deepen() (there are two - the static variable
           and the one in struct upload_pack_data).
       (3) Before deepen() computes the list of reachable shallows, it first
           needs to mark all "our" refs as OUR_REF. v2 currently does not do
           this, because unlike v0, it is not needed otherwise.
      
      Fix all this and include a test demonstrating that it works now. For
      (2), the static variable deepen_relative is also eliminated, removing a
      source of confusion.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Reviewed-by: 's avatarJosh Steadmon <steadmon@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      5056cf4a
    • Jonathan Tan's avatar
      fetch-pack: do not take shallow lock unnecessarily · bd0b42ae
      Jonathan Tan authored
      When fetching using protocol v2, the remote may send a "shallow-info"
      section if the client is shallow. If so, Git as the client currently
      takes the shallow file lock, even if the "shallow-info" section is
      empty.
      
      This is not a problem except that Git does not support taking the
      shallow file lock after modifying the shallow file, because
      is_repository_shallow() stores information that is never cleared. And
      this take-after-modify occurs when Git does a tag-following fetch from a
      shallow repository on a transport that does not support tag following
      (since in this case, 2 fetches are performed).
      
      To solve this issue, take the shallow file lock (and perform all other
      shallow processing) only if the "shallow-info" section is non-empty;
      otherwise, behave as if it were empty.
      
      A full solution (probably, ensuring that any action of committing
      shallow file locks also includes clearing the information stored by
      is_repository_shallow()) would solve the issue without need for this
      patch, but this patch is independently useful (as an optimization to
      prevent writing a file in an unnecessary case), hence why I wrote it. I
      have included a NEEDSWORK outlining the full solution.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      bd0b42ae
  10. 02 Jan, 2019 2 commits
    • Masaya Suzuki's avatar
      pack-protocol.txt: accept error packets in any context · 2d103c31
      Masaya Suzuki authored
      In the Git pack protocol definition, an error packet may appear only in
      a certain context. However, servers can face a runtime error (e.g. I/O
      error) at an arbitrary timing. This patch changes the protocol to allow
      an error packet to be sent instead of any packet.
      
      Without this protocol spec change, when a server cannot process a
      request, there's no way to tell that to a client. Since the server
      cannot produce a valid response, it would be forced to cut a connection
      without telling why. With this protocol spec change, the server can be
      more gentle in this situation. An old client may see these error packets
      as an unexpected packet, but this is not worse than having an unexpected
      EOF.
      
      Following this protocol spec change, the error packet handling code is
      moved to pkt-line.c. Implementation wise, this implementation uses
      pkt-line to communicate with a subprocess. Since this is not a part of
      Git protocol, it's possible that a packet that is not supposed to be an
      error packet is mistakenly parsed as an error packet. This error packet
      handling is enabled only for the Git pack protocol parsing code
      considering this.
      Signed-off-by: 's avatarMasaya Suzuki <masayasuzuki@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      2d103c31
    • Masaya Suzuki's avatar
      Use packet_reader instead of packet_read_line · 01f9ec64
      Masaya Suzuki authored
      By using and sharing a packet_reader while handling a Git pack protocol
      request, the same reader option is used throughout the code. This makes
      it easy to set a reader option to the request parsing code.
      Signed-off-by: 's avatarMasaya Suzuki <masayasuzuki@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      01f9ec64
  11. 13 Nov, 2018 1 commit
    • Jeff King's avatar
      fetch-pack: drop custom loose object cache · 97b2fa08
      Jeff King authored
      Commit 024aa469 (fetch-pack.c: use oidset to check existence of loose
      object, 2018-03-14) added a cache to avoid calling stat() for a bunch of
      loose objects we don't have.
      
      Now that OBJECT_INFO_QUICK handles this caching itself, we can drop the
      custom solution.
      
      Note that this might perform slightly differently, as the original code
      stopped calling readdir() when we saw more loose objects than there were
      refs. So:
      
        1. The old code might have spent work on readdir() to fill the cache,
           but then decided there were too many loose objects, wasting that
           effort.
      
        2. The new code might spend a lot of time on readdir() if you have a
           lot of loose objects, even though there are very few objects to
           ask about.
      
      In practice it probably won't matter either way; see the previous commit
      for some discussion of the tradeoff.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      97b2fa08
  12. 01 Nov, 2018 1 commit
    • Jonathan Tan's avatar
      fetch-pack: be more precise in parsing v2 response · 5400b2a2
      Jonathan Tan authored
      Each section in a protocol v2 response is followed by either a DELIM
      packet (indicating more sections to follow) or a FLUSH packet
      (indicating none to follow). But when parsing the "acknowledgments"
      section, do_fetch_pack_v2() is liberal in accepting both, but determines
      whether to continue reading or not based solely on the contents of the
      "acknowledgments" section, not on whether DELIM or FLUSH was read.
      
      There is no issue with a protocol-compliant server, but can result in
      confusing error messages when communicating with a server that
      serves unexpected additional sections. Consider a server that sends
      "new-section" after "acknowledgments":
      
       - client writes request
       - client reads the "acknowledgments" section which contains no "ready",
         then DELIM
       - since there was no "ready", client needs to continue negotiation, and
         writes request
       - client reads "new-section", and reports to the end user "expected
         'acknowledgments', received 'new-section'"
      
      For the person debugging the involved Git implementation(s), the error
      message is confusing in that "new-section" was not received in response
      to the latest request, but to the first one.
      
      One solution is to always continue reading after DELIM, but in this
      case, we can do better. We know from the protocol that "ready" means at
      least the packfile section is coming (hence, DELIM) and that no "ready"
      means that no sections are to follow (hence, FLUSH). So teach
      process_acks() to enforce this.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      5400b2a2
  13. 09 Oct, 2018 1 commit
    • Jeff King's avatar
      transport: drop refnames from for_each_alternate_ref · bdf4276c
      Jeff King authored
      None of the current callers use the refname parameter we pass to their
      callbacks. In theory somebody _could_ do so, but it's actually quite
      weird if you think about it: it's a ref in somebody else's repository.
      So the name has no meaning locally, and in fact there may be duplicates
      if there are multiple alternates.
      
      The users of this interface really only care about seeing some ref tips,
      since that promises that the alternate has the full commit graph
      reachable from there. So let's keep the information we pass back to the
      bare minimum.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarTaylor Blau <me@ttaylorr.com>
      Acked-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      bdf4276c
  14. 07 Oct, 2018 1 commit
  15. 04 Oct, 2018 4 commits
    • René Scharfe's avatar
      fetch-pack: load tip_oids eagerly iff needed · 22a16465
      René Scharfe authored
      tip_oids_contain() lazily loads refs into an oidset at its first call.
      It abuses the internal (sub)member .map.tablesize of that oidset to
      check if it has done that already.
      
      Determine if the oidset needs to be populated upfront and then do that
      instead.  This duplicates a loop, but simplifies the existing one by
      separating concerns between the two.
      Signed-off-by: 's avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      22a16465
    • René Scharfe's avatar
      fetch-pack: factor out is_unmatched_ref() · bf73282c
      René Scharfe authored
      Move the code to determine if a request is unmatched to its own little
      helper.  This allows us to reuse it in a subsequent patch.
      Signed-off-by: 's avatarRene Scharfe <l.s.r@web.de>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      bf73282c
    • Jonathan Tan's avatar
      fetch-pack: exclude blobs when lazy-fetching trees · 4c7f9567
      Jonathan Tan authored
      A partial clone with missing trees can be obtained using "git clone
      --filter=tree:none <repo>". In such a repository, when a tree needs to
      be lazily fetched, any tree or blob it directly or indirectly references
      is fetched as well, regardless of whether the original command required
      those objects, or if the local repository already had some of them.
      
      This is because the fetch protocol, which the lazy fetch uses, does not
      allow clients to request that only the wanted objects be sent, which
      would be the ideal solution. This patch implements a partial solution:
      specify the "blob:none" filter, somewhat reducing the fetch payload.
      
      This change has no effect when lazily fetching blobs (due to how filters
      work). And if lazily fetching a commit (such repositories are difficult
      to construct and is not a use case we support very well, but it is
      possible), referenced commits and trees are still fetched - only the
      blobs are not fetched.
      
      The necessary code change is done in fetch_pack() instead of somewhere
      closer to where the "filter" instruction is written to the wire so that
      only one part of the code needs to be changed in order for users of all
      protocol versions to benefit from this optimization.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      4c7f9567
    • Jonathan Tan's avatar
      fetch-pack: avoid object flags if no_dependents · 12f19a98
      Jonathan Tan authored
      When fetch_pack() is invoked as part of another Git command (due to a
      lazy fetch from a partial clone, for example), it uses object flags that
      may already be used by the outer Git command.
      
      The commit that introduced the lazy fetch feature (88e2f9ed
      ("introduce fetch-object: fetch one promisor object", 2017-12-05)) tried
      to avoid this overlap, but it did not avoid it totally. It was
      successful in avoiding writing COMPLETE, but did not avoid reading
      COMPLETE, and did not avoid writing and reading ALTERNATE.
      
      Ensure that no flags are written or read by fetch_pack() in the case
      where it is used to perform a lazy fetch. To do this, it is sufficient
      to avoid checking completeness of wanted refs (unnecessary in the case
      of lazy fetches), and to avoid negotiation-related work (in the current
      implementation, already, no negotiation is performed). After that was
      done, the lack of overlap was verified by checking all direct and
      indirect usages of COMPLETE and ALTERNATE - that they are read or
      written only if no_dependents is false.
      
      There are other possible solutions to this issue:
      
       (1) Split fetch-pack.{c,h} into a flag-using part and a non-flag-using
           part, and whenever no_dependents is set, only use the
           non-flag-using part.
       (2) Make fetch_pack() be able to be used with arbitrary repository
           objects. fetch_pack() should then create its own repository object
           based on the given repository object, with its own object
           hashtable, so that the flags do not conflict.
      
      (1) is possible but invasive - some functions would need to be split;
      and such invasiveness would potentially be unnecessary if we ever were
      to need (2) anyway. (2) would be useful if we were to support, say,
      submodules that were partial clones themselves, but I don't know when or
      if the Git project plans to support those.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      12f19a98
  16. 29 Aug, 2018 1 commit
    • Jeff King's avatar
      convert "oidcmp() != 0" to "!oideq()" · 9001dc2a
      Jeff King authored
      This is the flip side of the previous two patches: checking
      for a non-zero oidcmp() can be more strictly expressed as
      inequality. Like those patches, we write "!= 0" in the
      coccinelle transformation, which covers by isomorphism the
      more common:
      
        if (oidcmp(E1, E2))
      
      As with the previous two patches, this patch can be achieved
      almost entirely by running "make coccicheck"; the only
      differences are manual line-wrap fixes to match the original
      code.
      
      There is one thing to note for anybody replicating this,
      though: coccinelle 1.0.4 seems to miss the case in
      builtin/tag.c, even though it's basically the same as all
      the others. Running with 1.0.7 does catch this, so
      presumably it's just a coccinelle bug that was fixed in the
      interim.
      Signed-off-by: 's avatarJeff King <peff@peff.net>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      9001dc2a
  17. 01 Aug, 2018 1 commit
    • Jonathan Tan's avatar
      fetch-pack: unify ref in and out param · e2842b39
      Jonathan Tan authored
      When a user fetches:
       - at least one up-to-date ref and at least one non-up-to-date ref,
       - using HTTP with protocol v0 (or something else that uses the fetch
         command of a remote helper)
      some refs might not be updated after the fetch.
      
      This bug was introduced in commit 989b8c44 ("fetch-pack: put shallow
      info in output parameter", 2018-06-28) which allowed transports to
      report the refs that they have fetched in a new out-parameter
      "fetched_refs". If they do so, transport_fetch_refs() makes this
      information available to its caller.
      
      Users of "fetched_refs" rely on the following 3 properties:
       (1) it is the complete list of refs that was passed to
           transport_fetch_refs(),
       (2) it has shallow information (REF_STATUS_REJECT_SHALLOW set if
           relevant), and
       (3) it has updated OIDs if ref-in-want was used (introduced after
           989b8c44).
      
      In an effort to satisfy (1), whenever transport_fetch_refs()
      filters the refs sent to the transport, it re-adds the filtered refs to
      whatever the transport supplies before returning it to the user.
      However, the implementation in 989b8c44 unconditionally re-adds the
      filtered refs without checking if the transport refrained from reporting
      anything in "fetched_refs" (which it is allowed to do), resulting in an
      incomplete list, no longer satisfying (1).
      
      An earlier effort to resolve this [1] solved the issue by readding the
      filtered refs only if the transport did not refrain from reporting in
      "fetched_refs", but after further discussion, it seems that the better
      solution is to revert the API change that introduced "fetched_refs".
      This API change was first suggested as part of a ref-in-want
      implementation that allowed for ref patterns and, thus, there could be
      drastic differences between the input refs and the refs actually fetched
      [2]; we eventually decided to only allow exact ref names, but this API
      change remained even though its necessity was decreased.
      
      Therefore, revert this API change by reverting commit 989b8c44, and
      make receive_wanted_refs() update the OIDs in the sought array (like how
      update_shallow() updates shallow information in the sought array)
      instead. A test is also included to show that the user-visible bug
      discussed at the beginning of this commit message no longer exists.
      
      [1] https://public-inbox.org/git/20180801171806.GA122458@google.com/
      [2] https://public-inbox.org/git/86a128c5fb710a41791e7183207c4d64889f9307.1485381677.git.jonathantanmy@google.com/Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      e2842b39
  18. 27 Jul, 2018 1 commit
    • Ævar Arnfjörð Bjarmason's avatar
      fetch: implement fetch.fsck.* · 1362df0d
      Ævar Arnfjörð Bjarmason authored
      Implement support for fetch.fsck.* corresponding with the existing
      receive.fsck.*. This allows for pedantically cloning repositories with
      specific issues without turning off fetch.fsckObjects.
      
      One such repository is https://github.com/robbyrussell/oh-my-zsh.git
      which before this change will emit this error when cloned with
      fetch.fsckObjects:
      
          error: object 2b7227859263b6aabcc28355b0b994995b7148b6: zeroPaddedFilemode: contains zero-padded file modes
          fatal: Error in object
          fatal: index-pack failed
      
      Now with fetch.fsck.zeroPaddedFilemode=warn we'll warn about that
      issue, but the clone will succeed:
      
          warning: object 2b7227859263b6aabcc28355b0b994995b7148b6: zeroPaddedFilemode: contains zero-padded file modes
          warning: object a18c4d13c2a5fa2d4ecd5346c50e119b999b807d: zeroPaddedFilemode: contains zero-padded file modes
          warning: object 84df066176c8da3fd59b13731a86d90f4f1e5c9d: zeroPaddedFilemode: contains zero-padded file modes
      
      The motivation for this is to be able to turn on fetch.fsckObjects
      globally across a fleet of computers but still be able to manually
      clone various legacy repositories by either white-listing specific
      issues, or better yet whitelist specific objects.
      
      The use of --git-dir=* instead of -C in the tests could be considered
      somewhat archaic, but the tests I'm adding here are duplicating the
      corresponding receive.* tests with as few changes as possible.
      Signed-off-by: Ævar Arnfjörð Bjarmason's avatarÆvar Arnfjörð Bjarmason <avarab@gmail.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      1362df0d
  19. 23 Jul, 2018 1 commit
  20. 16 Jul, 2018 1 commit
    • Jonathan Tan's avatar
      negotiator/skipping: skip commits during fetch · 42cc7485
      Jonathan Tan authored
      Introduce a new negotiation algorithm used during fetch that skips
      commits in an effort to find common ancestors faster. The skips grow
      similarly to the Fibonacci sequence as the commit walk proceeds further
      away from the tips. The skips may cause unnecessary commits to be
      included in the packfile, but the negotiation step typically ends more
      quickly.
      
      Usage of this algorithm is guarded behind the configuration flag
      fetch.negotiationAlgorithm.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      42cc7485
  21. 03 Jul, 2018 2 commits
    • Jonathan Tan's avatar
      fetch-pack: support negotiation tip whitelist · 3390e42a
      Jonathan Tan authored
      During negotiation, fetch-pack eventually reports as "have" lines all
      commits reachable from all refs. Allow the user to restrict the commits
      sent in this way by providing a whitelist of tips; only the tips
      themselves and their ancestors will be sent.
      
      Both globs and single objects are supported.
      
      This feature is only supported for protocols that support connect or
      stateless-connect (such as HTTP with protocol v2).
      
      This will speed up negotiation when the repository has multiple
      relatively independent branches (for example, when a repository
      interacts with multiple repositories, such as with linux-next [1] and
      torvalds/linux [2]), and the user knows which local branch is likely to
      have commits in common with the upstream branch they are fetching.
      
      [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/
      [2] https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      3390e42a
    • Jonathan Tan's avatar
      fetch-pack: write shallow, then check connectivity · cf1e7c07
      Jonathan Tan authored
      When fetching, connectivity is checked after the shallow file is
      updated. There are 2 issues with this: (1) the connectivity check is
      only performed up to ancestors of existing refs (which is not thorough
      enough if we were deepening an existing ref in the first place), and (2)
      there is no rollback of the shallow file if the connectivity check
      fails.
      
      To solve (1), update the connectivity check to check the ancestry chain
      completely in the case of a deepening fetch by refraining from passing
      "--not --all" when invoking rev-list in connected.c.
      
      To solve (2), have fetch_pack() perform its own connectivity check
      before updating the shallow file. To support existing use cases in which
      "git fetch-pack" is used to download objects without much regard as to
      the connectivity of the resulting objects with respect to the existing
      repository, the connectivity check is only done if necessary (that is,
      the fetch is not a clone, and the fetch involves shallow/deepen
      functionality). "git fetch" still performs its own connectivity check,
      preserving correctness but sometimes performing redundant work. This
      redundancy is mitigated by the fact that fetch_pack() reports if it has
      performed a connectivity check itself, and if the transport supports
      connect or stateless-connect, it will bubble up that report so that "git
      fetch" knows not to perform the connectivity check in such a case.
      
      This was noticed when a user tried to deepen an existing repository by
      fetching with --no-shallow from a server that did not send all necessary
      objects - the connectivity check as run by "git fetch" succeeded, but a
      subsequent "git fsck" failed.
      Signed-off-by: 's avatarJonathan Tan <jonathantanmy@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      cf1e7c07
  22. 29 Jun, 2018 4 commits
  23. 28 Jun, 2018 2 commits
    • Brandon Williams's avatar
      fetch-pack: implement ref-in-want · 73302051
      Brandon Williams authored
      Implement ref-in-want on the client side so that when a server supports
      the "ref-in-want" feature, a client will send "want-ref" lines for each
      reference the client wants to fetch.  This feature allows clients to
      tolerate inconsistencies that exist when a remote repository's refs
      change during the course of negotiation.
      
      This allows a client to request to request a particular ref without
      specifying the OID of the ref.  This means that instead of hitting an
      error when a ref no longer points at the OID it did at the beginning of
      negotiation, negotiation can continue and the value of that ref will be
      sent at the termination of negotiation, just before a packfile is sent.
      
      More information on the ref-in-want feature can be found in
      Documentation/technical/protocol-v2.txt.
      Signed-off-by: 's avatarBrandon Williams <bmwill@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      73302051
    • Brandon Williams's avatar
      fetch-pack: put shallow info in output parameter · 989b8c44
      Brandon Williams authored
      Expand the transport fetch method signature, by adding an output
      parameter, to allow transports to return information about the refs they
      have fetched.  Then communicate shallow status information through this
      mechanism instead of by modifying the input list of refs.
      
      This does require clients to sometimes generate the ref map twice: once
      from the list of refs provided by the remote (as is currently done) and
      potentially once from the new list of refs that the fetch mechanism
      provides.
      Signed-off-by: 's avatarBrandon Williams <bmwill@google.com>
      Signed-off-by: 's avatarJunio C Hamano <gitster@pobox.com>
      989b8c44
  24. 15 Jun, 2018 3 commits