Skip to content
  • Thomas Gummerer's avatar
    linear-assignment: fix potential out of bounds memory access · e467a90c
    Thomas Gummerer authored and Junio C Hamano's avatar Junio C Hamano committed
    Currently the 'compute_assignment()' function may read memory out
    of bounds, even if used correctly.  Namely this happens when we only
    have one column.  In that case we try to calculate the initial
    minimum cost using '!j1' as column in the reduction transfer code.
    That in turn causes us to try and get the cost from column 1 in the
    cost matrix, which does not exist, and thus results in an out of
    bounds memory read.
    
    In the original paper [1], the example code initializes that minimum
    cost to "infinite".  We could emulate something similar by setting the
    minimum cost to INT_MAX, which would result in the same minimum cost
    as the current algorithm, as we'd always go into the if condition at
    least once, except when we only have one column, and column_count thus
    equals 1.
    
    If column_count does equal 1, the condition in the loop would always
    be false, and we'd end up with a minimum of INT_MAX, which may lead to
    integer overflows later in the algorithm...
    e467a90c