Move model selection handling to a middleware
Problem to solve
To fetch which model should be called for a request, each endpoint implements their own way of handling model information input. For example, v2/suggestions/generation requires model_identifier, while v2/agents requires identifier, and v3/suggestions/generation doesn't even support model configuration support.
Proposal
Make model selection transparent from the endpoint logic by moving it to a Starllete middleware, that already parses the payload and injects the correct model into the scope. This way, all endpoints (old and new) are have automatic support for model selection.
Links / references
- AIGW request middlewares - https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/blob/main/ai_gateway/api/middleware.py#L435-L434
- Starlette middleware documentation https://www.starlette.io/middleware/
Edited by Eduardo Bonet