Add support for chunking strategies
We need to support chunking during indexing and retrieval. For example, we need to chunk large texts into smaller texts by size or chunk a code blob into methods.
-
Build a way to choose a chunking strategy for a collection -
Use the strategy for indexing -
Create a default chunking strategy which does nothing -
Figure out how to deal with multiple chunks per "document" and how that works for retrieval
Edited by Madelein van Niekerk