Skip to content

feat(datastore): estimated namespace size with timeout awareness

João Pereira requested to merge 853-api into master

Related to #853 (closed).

This MR introduces a new query to estimate the size of top-level namespaces, behind the EstimatedSizeWithDescendants repository store method. This will be used as a fallback from the main SizeWithDescendants method in case of timeout.

To avoid repeated consecutive failures in executing SizeWithDescendants (these are resource-intensive and time consuming), we record a special key in Redis whenever this query times out for a given repository. This key is set to expire after 24h. If the key exists, SizeWithDescendants will exit early with a special error. This will sign the caller (the corresponding API change will come in the next MR) that it should use EstimatedSizeWithDescendants instead.

I've considered calling estimatedSizeWithDescendants automatically from within SizeWithDescendants, abstracting this away from the caller, but: 1) it got really hard to test 2) it restricts other possible use cases by not giving the caller the ability to orchestrate it (such as using specific timeouts for each method). So I end up going this way to make it composable and simpler to test.

Edited by João Pereira

Merge request reports