• Jeff King's avatar
    xdiff: reject files larger than ~1GB · dcd1742e
    Jeff King authored
    The xdiff code is not prepared to handle extremely large
    files. It uses "int" in many places, which can overflow if
    we have a very large number of lines or even bytes in our
    input files. This can cause us to produce incorrect diffs,
    with no indication that the output is wrong. Or worse, we
    may even underallocate a buffer whose size is the result of
    an overflowing addition.
    We're much better off to tell the user that we cannot diff
    or merge such a large file. This patch covers both cases,
    but in slightly different ways:
      1. For merging, we notice the large file and cleanly fall
         back to a binary merge (which is effectively "we cannot
         merge this").
      2. For diffing, we make the binary/text distinction much
         earlier, and in many different places. For this case,
         we'll use the xdi_diff as our choke point, and reject
         any diff there before it hits the xdiff code.
         This means in most cases we'll die() immediately after.
         That's not ideal, but in practice we shouldn't
         generally hit this code path unless the user is trying
         to do something tricky. We already consider files
         larger than core.bigfilethreshold to be binary, so this
         code would only kick in when that is circumvented
         (either by bumping that value, or by using a
         .gitattribute to mark a file as diffable).
         In other words, we can avoid being "nice" here, because
         there is already nice code that tries to do the right
         thing. We are adding the suspenders to the nice code's
         belt, so notice when it has been worked around (both to
         protect the user from malicious inputs, and because it
         is better to die() than generate bogus output).
    The maximum size was chosen after experimenting with feeding
    large files to the xdiff code. It's just under a gigabyte,
    which leaves room for two obvious cases:
      - a diff3 merge conflict result on files of maximum size X
        could be 3*X plus the size of the markers, which would
        still be only about 3G, which fits in a 32-bit int.
      - some of the diff code allocates arrays of one int per
        record. Even if each file consists only of blank lines,
        then a file smaller than 1G will have fewer than 1G
        records, and therefore the int array will fit in 4G.
    Since the limit is arbitrary anyway, I chose to go under a
    gigabyte, to leave a safety margin (e.g., we would not want
    to overflow by allocating "(records + 1) * sizeof(int)" or
    Signed-off-by: default avatarJeff King <peff@peff.net>
    Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
xdiff-interface.h 1.14 KB