Preprocess pipeline
Add a bulk preprocess step between reading references from the queue and processing each reference.
The current pipeline allows for preloading database records to prevent N+1 but this should be extended to allow for chaining of multiple preprocessing steps.
Examples include:
- preloading database records
- bulk embedding generation
- bulk summary generation
Every reference class should specify how (if any) to preprocess data.
Edited by Madelein van Niekerk