Infinite Scroll Issues
I think that infinite scroll (querying for the next X amount of items as you scroll down and loading new elements) isn't the best way to load libraries. Not only is it poor UX and doesn't give a complete picture of a library but it also will break as soon as sorting or filtering is implemented. I know a lot of stuff was implemented with the assumption people know what is in the library but once a new user gets invited or you get invited to a different server (in the case of federation), if you aren't familiar with the content browsing for unknown content is a chore.
I think the best option is to load empty elements with "loading" text and placeholder image. It may not be the most performant in terms of loading initially but will vastly improve UX.