Skip to content

Move model selection handling to a middleware

Problem to solve

To fetch which model should be called for a request, each endpoint implements their own way of handling model information input. For example, v2/suggestions/generation requires model_identifier, while v2/agents requires identifier, and v3/suggestions/generation doesn't even support model configuration support.

Proposal

Make model selection transparent from the endpoint logic by moving it to a Starllete middleware, that already parses the payload and injects the correct model into the scope. This way, all endpoints (old and new) are have automatic support for model selection.

Links / references

Edited by Eduardo Bonet