Skip to content

Safer upload algorithm

Luke Champine requested to merge upload into master

The old upload algorithm was fast, but could result in a dangerously skewed distribution of pieces if one host was much faster than the others (e.g. a host on the local network). The new algorithm distributes pieces evenly, but is bottlenecked by slow hosts.

As with the download test, I have also introduced random (20%) upload failures. Old:

--- PASS: TestErasureUpload (0.51s)
    upload_test.go:95: Host #: 0    Delay: 1ms      # Pieces: 343 (100x faster)
    upload_test.go:95: Host #: 1    Delay: 100ms    # Pieces: 0   (always fails)
    upload_test.go:95: Host #: 2    Delay: 100ms    # Pieces: 3
    upload_test.go:95: Host #: 3    Delay: 100ms    # Pieces: 3
    upload_test.go:95: Host #: 4    Delay: 100ms    # Pieces: 4
    upload_test.go:95: Host #: 5    Delay: 100ms    # Pieces: 3
    upload_test.go:95: Host #: 6    Delay: 100ms    # Pieces: 4
    upload_test.go:95: Host #: 7    Delay: 100ms    # Pieces: 4
    upload_test.go:95: Host #: 8    Delay: 100ms    # Pieces: 2
    upload_test.go:95: Host #: 9    Delay: 100ms    # Pieces: 4
    upload_test.go:95: Host #: 10   Delay: 100ms    # Pieces: 3
    upload_test.go:95: Host #: 11   Delay: 100ms    # Pieces: 5

New:

--- PASS: TestErasureUpload (3.98s)
    upload_test.go:95: Host #: 0    Delay: 1ms      # Pieces: 34 (100x faster)
    upload_test.go:95: Host #: 1    Delay: 100ms    # Pieces: 0  (always fails)
    upload_test.go:95: Host #: 2    Delay: 100ms    # Pieces: 29
    upload_test.go:95: Host #: 3    Delay: 100ms    # Pieces: 30
    upload_test.go:95: Host #: 4    Delay: 100ms    # Pieces: 34
    upload_test.go:95: Host #: 5    Delay: 100ms    # Pieces: 35
    upload_test.go:95: Host #: 6    Delay: 100ms    # Pieces: 34
    upload_test.go:95: Host #: 7    Delay: 100ms    # Pieces: 30
    upload_test.go:95: Host #: 8    Delay: 100ms    # Pieces: 29
    upload_test.go:95: Host #: 9    Delay: 100ms    # Pieces: 32
    upload_test.go:95: Host #: 10   Delay: 100ms    # Pieces: 32
    upload_test.go:95: Host #: 11   Delay: 100ms    # Pieces: 33

(Fun fact: the old algo deadlocked if any piece failed to upload!! Who would write such unstable code? :P)

Merge request reports