feat(datastore): estimated namespace size with timeout awareness
Related to #853 (closed).
This MR introduces a new query to estimate the size of top-level namespaces, behind the EstimatedSizeWithDescendants
repository store method. This will be used as a fallback from the main SizeWithDescendants
method in case of timeout.
To avoid repeated consecutive failures in executing SizeWithDescendants
(these are resource-intensive and time consuming), we record a special key in Redis whenever this query times out for a given repository. This key is set to expire after 24h. If the key exists, SizeWithDescendants
will exit early with a special error. This will sign the caller (the corresponding API change will come in the next MR) that it should use EstimatedSizeWithDescendants
instead.
I've considered calling estimatedSizeWithDescendants
automatically from within SizeWithDescendants
, abstracting this away from the caller, but: 1) it got really hard to test 2) it restricts other possible use cases by not giving the caller the ability to orchestrate it (such as using specific timeouts for each method). So I end up going this way to make it composable and simpler to test.