Better download algorithm
In short: download pieces in serial. For each chunk, pick a random host who has a piece from that chunk and try to download it. If the download fails, try the next host. Repeat until you have the minimum number of pieces required for download.
I also introduced random download failures in the testHost
, to better simulate real-world conditions. The test is run with a 20% failure rate (per piece).
Comparison of old algo to new: Old:
--- PASS: TestErasureDownload (3.91s)
download_test.go:117: Host 0: Fetched: 37/39
download_test.go:117: Host 1: Fetched: 0/39 (always fails)
download_test.go:117: Host 2: Fetched: 33/39
download_test.go:117: Host 3: Fetched: 32/39
download_test.go:117: Host 4: Fetched: 31/39
download_test.go:117: Host 5: Fetched: 33/39
download_test.go:117: Host 6: Fetched: 34/39
download_test.go:117: Host 7: Fetched: 30/39
download_test.go:117: Host 8: Fetched: 30/39
download_test.go:117: Host 9: Fetched: 29/39
download_test.go:117: Host 10: Fetched: 32/39
download_test.go:117: Host 11: Fetched: 29/39
download_test.go:121: Optimal fetches: 78
download_test.go:122: Total fetches: 468
New:
--- PASS: TestErasureDownload (1.11s)
download_test.go:117: Host 0: Fetched: 5/10
download_test.go:117: Host 1: Fetched: 0/10 (always fails)
download_test.go:117: Host 2: Fetched: 4/5
download_test.go:117: Host 3: Fetched: 7/9
download_test.go:117: Host 4: Fetched: 8/11
download_test.go:117: Host 5: Fetched: 4/7
download_test.go:117: Host 6: Fetched: 3/6
download_test.go:117: Host 7: Fetched: 7/9
download_test.go:117: Host 8: Fetched: 8/8
download_test.go:117: Host 9: Fetched: 15/15
download_test.go:117: Host 10: Fetched: 8/9
download_test.go:117: Host 11: Fetched: 9/12
download_test.go:121: Optimal fetches: 78
download_test.go:122: Total fetches: 111