Skip to content

refactor(storage): changed upload purging algorithm to reduce memory usage for very large registries

Related to #216

Implementation of "Pipelines" solution from #216 issue.

I changed getOutstandingUploads return signature to work with a channel of results/errors (outstandingUpload struct) which causes tiny test and PurgeUploads changes. Errors and results channels were combined in one results channel which made it possible to simplify inner fetching logic. I preserve syncMap, but added delete method to free used space as soon as possible. Also I delete uploadData constructor because we don't need pre-set startedAt field because by new logic we don't send uploadData with zero startedAt field.

Still think this approach is worse than "Database" described in #216. Maybe if i understand the database data structure better, a can go back and try to implement "Database" approach

Edited by Alex Komrakov

Merge request reports