Nightly CI + Nightly Builds
FEATURE REQUEST
Update our build process so that we run the full test suite every night (vlong tests included). I believe it makes sense at the same time to produce nightly releases
Description of Request
The siatest renter package will sometimes time out, especially now that we've added a boatload of tests. For the most part, these tests don't check any of the code that has changed, and so many of the more expensive tests don't really need to be run every commit. That doesn't mean we can just stop using the tests though.
To fix the timeouts, we should be more liberally labeling some tests as vlong, as vlong tests don't generally run in the CI.
At the same time, the vlong tests often don't get run at all, which is not great. Especially considering all of the NDFs and flakyness we've seen, and the small handful of production bugs this has caught on occasion, we should have a system in place to be regularly running the vlong tests. I think nightly builds make sense, because that means the vlong test suite can take up to several hours and it's not a big deal.
I could be wrong, and if I am this second piece should be moved to an entirely separate issue, but it makes sense to me that when we set up nightly CI, we should simultaneously set up nightly builds. If the CI passes (and only if the CI passes), a nightly build should be created that can be downloaded by users who want features which haven't been through a full release cycle yet, but also don't want to build the master branch themselves. We can then have either a webpage or some other link available somewhere that points to the most recent successful nightly build.
Design / Proposal
TBD