RefCounter: Actual Counter Values [please review design, so development can start]
Goal
The goal of this issue is to outline how we are going to implement the mechanism of incrementing/decrementing counter values in accordance to relevant events. We should agree on a strategy before starting to implement it.
Relevant events
- SiaFile deletion
- Other SiaFile events, such as
GrowNumChunks
,RemoveLastChunk
,AddPiece
- Backup creation
- Backup deletion (not implemented, yet)
SiaFile deletion
Deleting a SiaFile should result in decreasing all counters that belong to sectors of the SiaFile by one. As we don't have a mapping SiaFile <-> contract, we will need to scan all active contracts and then check each of them for roots that belong to the SiaFile being deleted. (Alternatively, we can implement the mentioned mapping.) This is an expensive disk IO operation that can take some time. This is problematic for performance because this operation is sometimes called in a loop.
Execution tree: SiaFile.Delete
->FileNode.managedDelete
->DirNode.managedDeleteFile
->FileSystem.managedDeleteFile
->FileSystem.DeleteFile
->Renter.DeleteFile
.
Renter.DeleteFile
is called by Renter.DeleteFiles
and Renter.threadedSynchronizeSnapshots
.
If the renter is deleting multiple files in a single DeleteFiles
operation it would be very inefficient to scan all contracts every time a file is deleted. It would make much more sense to compile a list of deleted roots and scan the contracts just once, searching for all roots at once. This can be organised in some sort of a cleanup
operation to be executed after the deletion.
Cleanup
We can trigger the cleanup in two ways:
- manual trigger by the party invoking the file deletion
- automatic trigger by a subsystem keeping track
Manual trigger
We can add a parameter to all deletion functions which instructs them whether to trigger a cleanup or not. This will give the caller the flexibility to request a cleanup only after they've completed the entire batch of actions they need to perform. This would also allow us to give control over the cleanup calls to exactly one party in the execution chain - once someone takes responsibility of calling the cleanup they can instruct everyone in the chain below them to not worry about it; alternatively, if they delegate that responsibility to the methods below them it would be up to those methods to either trigger the cleanup or delegate further down - depending on what makes sense.
Automatic trigger
Triggering the cleanup automatically can be achieved by either queuing up cleanup events and performing them via a cronjob (ticker) (potentially omitting the ones which are too new, e.g. <1 second old) or by spinning up a goroutine that would wait for all changes to the siafile to be over, sleeping for a bit, and then executing the cleanup.
Personally, I like the cronjob idea because we can expand it in the future and let it run in a completely decoupled way from the hot path serving requests.
Cleanup and deletions
Given that the actions triggered by a counter reaching zero are not implemented yet, we can include them in this cleanup
step. This will give us confidence that a counter temporarily reaching zero for a moment will not be GC-ed before we have a chance to increment its value again. An example scenario in this case would be a user triggering a "backup and delete" operation in which they create a backup and immediately delete their local files after that. In this situation we should ensure that all backed up sectors are preserved because of the backup.
Part of #3836