Refactor AIGateway::Client (and the other service clients)

Problem

The AiGateway::Client as of today is in a bad shape that it doesn't correctly reflect the interfaces of AI Gateway. It's tailored only for /v1/chat/agent and now it's spilling the business logic as endpoint_url parameter. For instance, ::Gitlab::Llm::AiGateway::Client::DEFAULT_INSTANT_MODEL and Gitlab::Llm::AiGateway::Client::ConnectionError doesn't make sense at all as it varies per feature. This is something we need to follow-up to correct the client interface.

From client perspective, their concern is which services, models or features to access, and low-level stuff like authentication and connection protocols should be abstracted under the client libraries. Like demonstrated in typical client libraries (e.g. Google Cloud libraries) and AiGateway::DocsClient, we should introduce a new client per service. This means we should have DuoChatClient, DocsSearchClient, VertexAIProxyClient, AnthrpicProxyClient etc instead of the single AiGateway::Client class. It should be relatively straight-forward to have the shared module for handling AI Gateway specific low-level stuff and plug it into these clients. Imagine, as a client, you would want to use a "BigQuery" service to create a dataset, not "Google Cloud Platform" to create a dataset (As that's too broad, you wouldn't have idea in which service the dataset will be created).

Proposal

One idea would be to have a corresponding client class to the cloud connector services. So duo_chat service => DuoChatClient#completion/#stream/#get_docs etc.

pseudo code:

# ee/lib/gitlab/duo/services/code_suggestions/client.rb

module Gitlab
  module Duo
    module Services
      module CodeSuggestions
        include Gitlab::Duo::Concerns::AiGateway # plugin AI Gateway specific concerns

        class Client
          def get_access_token
            # business logic
          end

          private

          # This should go to Cloud Connectoer specific concern
          def cloud_connector_service
            ::CloudConnector::AvailableServices.find_by_name(service_name)
          end

          def service_name
            self.class.name.split('::')[-2].underscore
          end

          def access_token
            cloud_connector_service.access_token(user)
          end
        end
      end
    end
  end
end

Refactor AIGateway::Client (and the other service clients)

Problem

Proposal

Further reading