Replicate existing permissions in main db to PGVector database
Leverage existing permissions and access controls in advanced search
Keyword search
Keyword retrievers for all required document types
Keyword result ranking
Get to BM25-level relevance/ranking/precision
Get for free when data is stored in Elastic
Vector search
Retrievers for all required embedded document types
Embed required documents into Elastic
Hybrid search
Combine result sets into one and normalize rankings
Leverage native hybrid search query endpoint so that a kNN query is combined with the current keyword query and use native reciprocal rank fusion to rerank and get the most relevant results
Every Elastic cluster needs to be on version 8.12+ and have a license.
Metadata filters
Ingest pipeline for all required filterable/sortable fields
Get for free when vectors are stored in Elastic
Experimentation & evaluation work
Analysis on the amount of resources required to store embeddings
Chunk size relevance testing and validation
Keyword search ranking experimentation
Hybrid search rank testing and validation
Was this result accurate? Did I expect to see it in this position?
Response time
Precision and other relevance metrics
Analysis on the amount of resources required to store embeddings
Chunk size relevance testing and validation
Hybrid search rank tuning
Was this result accurate? Did I expect to see it in this position?
Response time
Precision and other relevance metrics
Iteration 2
Feature
PG
ES
AI Reranking
Send hybrid search results to reranker
Work to host reranking model/AI Gateway
Send hybrid search results to reranker
Leverage native model hosting? OR:
Work to host reranking model/AI Gateway
Recursive retrieval
Determine the best way to chunk and store data in postgres * Implement recursive retrieval
Leverage native ability to store nested chunks
Implement recursive retrieval
Small-to-Big retrieval
Determine the best way to chunk and store data in postgres
Implement recursive retrieval
Leverage native ability to store nested chunks
Embedded Tables
Table ingestion and embedding pipeline
Table ingestion and embedding pipeline
Experimentation & evaluation work
Compare candidate reranking models for best results
Test and validate response relevance and accuracy for recursive retrieval
Test and validate response relevance and accuracy for small-to-big retrieval
Compare candidate reranking models for best results
Test and validate response relevance and accuracy for recursive retrieval
Test and validate response relevance and accuracy for small-to-big retrieval