Improve API access to a repository tree
Problem
Repository Tree REST API and as well as PaginatedTree GraphQL query enforce a pagination maximum of 100 elements.
For WebIDE case, we need to return the full list of folders and files of the repository in a performant way.
The Web IDE needs to support things like fuzzy file finding and moving directories without opening them. There's a number of operations that become unusable when we don't know the entire list of blobs up front. source
We added an option to skip pagination for this endpoint. This solution works, but it still feels like a hack. However, with the pagination it will require us to send thousands of requests (with 100 elements limit) to fetch all files. That's a big performance hit.
Moreover, the pagination for this endpoint does not make much sense from the performance point of view. We currently apply pagination after git ls-tree command was executed. In other words, we fetch the whole tree for paginated request first and only after that apply the pagination.
Goal
Investigate and find a way to deliver repository tree to WebIDE in an efficient way