Download Code
MERGE REQUEST
Overview
This MR enables the new download code and removes the old one.
The new DL code is based on an algorithm that tries to find the sweet spot of when to launch workers, or when to hold off a little bit because there might be a worker that resolves in the meantime that leads to a more optimal download. It does this using chimera workers, these workers are a combination of unresolved workers that together provide some counterweight to the resolved workers. The DL algorithm might favor a chimera worker over a resolved worker, causing the algorithm to not launch a worker for the time being.
We do not only optimize for speed, but also for cost, and we do this using the price per millisecond variable. When we find a worker set that's viable, meaning that its chance of completing within the set's duration is greater than 50%, we will try and replace workers with cheaper workers. Worker sets are compared on the basis of a scoring function that uses the price per millisecond (ppms) and the worker set cost to calculate a penalty added to the worker set duration. Lowering the ppms essentially favors launching fast workers sooner.
There is also the concept of a base cost, which serves as a protection against cheap (or free) hosts. By adding a base bandwidth cost to all downloads, we essentially ensure super cheap workers are not necessarily favored by the download algorithm. Without this precaution, a single host might influence the overall performance of downloads. The base cost also directly influences the cost of an overdrive worker, by increasing the base cost the cost to overdrive increases along with it.
We are going to deploy with the following production settings:
- 200 SC/TB base cost
- 40 nS price per millisecond
- 50% chance
Benchmarks
baseline: https://siasky.net/CABEZmFVxRnHdgu4wZXyF_77P3fBkt_yxm9TcUxYsPyydg
dl code: https://siasky.net/CADcud0u-4txHcEvUaca_OLHsR2YU91K4ARcLkUHU1Dlyw
Example for Visual Changes
N/A
Checklist
Review and complete the checklist to ensure that the MR is complete before assigned to an approver.
-
All new methods or updated methods have clear docstrings -
Testing added or updated for new methods -
Any new packages are added to Makefile and .gitlab-ci.yml -
API documentation updated for API updates -
Module README.md updated for changes to workflow -
Changelog File Created
Issues Closed
N/A