Add support for chunking strategies

We need to support chunking during indexing and retrieval. For example, we need to chunk large texts into smaller texts by size or chunk a code blob into methods.

  • Build a way to choose a chunking strategy for a collection
  • Use the strategy for indexing
  • Create a default chunking strategy which does nothing
  • Figure out how to deal with multiple chunks per "document" and how that works for retrieval
Edited by Madelein van Niekerk