Skip to content

Better download algorithm

Luke Champine requested to merge download into master

In short: download pieces in serial. For each chunk, pick a random host who has a piece from that chunk and try to download it. If the download fails, try the next host. Repeat until you have the minimum number of pieces required for download.

I also introduced random download failures in the testHost, to better simulate real-world conditions. The test is run with a 20% failure rate (per piece).

Comparison of old algo to new: Old:

--- PASS: TestErasureDownload (3.91s)
    download_test.go:117: Host  0:  Fetched: 37/39
    download_test.go:117: Host  1:  Fetched: 0/39 (always fails)
    download_test.go:117: Host  2:  Fetched: 33/39
    download_test.go:117: Host  3:  Fetched: 32/39
    download_test.go:117: Host  4:  Fetched: 31/39
    download_test.go:117: Host  5:  Fetched: 33/39
    download_test.go:117: Host  6:  Fetched: 34/39
    download_test.go:117: Host  7:  Fetched: 30/39
    download_test.go:117: Host  8:  Fetched: 30/39
    download_test.go:117: Host  9:  Fetched: 29/39
    download_test.go:117: Host 10:  Fetched: 32/39
    download_test.go:117: Host 11:  Fetched: 29/39
    download_test.go:121: Optimal fetches: 78
    download_test.go:122: Total fetches:   468

New:

--- PASS: TestErasureDownload (1.11s)
    download_test.go:117: Host  0:  Fetched: 5/10
    download_test.go:117: Host  1:  Fetched: 0/10 (always fails)
    download_test.go:117: Host  2:  Fetched: 4/5
    download_test.go:117: Host  3:  Fetched: 7/9
    download_test.go:117: Host  4:  Fetched: 8/11
    download_test.go:117: Host  5:  Fetched: 4/7
    download_test.go:117: Host  6:  Fetched: 3/6
    download_test.go:117: Host  7:  Fetched: 7/9
    download_test.go:117: Host  8:  Fetched: 8/8
    download_test.go:117: Host  9:  Fetched: 15/15
    download_test.go:117: Host 10:  Fetched: 8/9
    download_test.go:117: Host 11:  Fetched: 9/12
    download_test.go:121: Optimal fetches: 78
    download_test.go:122: Total fetches:   111

Merge request reports