spike: Use LiteLLM as backend to connect to Model providers
Problem to solve
We need to add logic per model provider to call them from AI Gateway https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/tree/main/ai_gateway/models?ref_type=heads, and if we want to scale the number of LLMs that would be a lot of work to maintain.
Proposal
Use LiteLLM - https://github.com/BerriAI/litellm as backend for models, it offers support to a multitude off LLMs out of the box (see compatibility matrix)
Further details
- LiteLLM can be plugged as a library, there's no need to run a service for it
- Support for new models come faster through open source contributions