Support self-signed certificates in AI Gateway

Problem to solve

Currently, AI Gateway (AIGW) does not properly handle self-signed certificates when connecting to GitLab instances, resulting in SSL verification errors.

Current Behaviour

AIGW fails with [SSL: CERTIFICATE_VERIFY_FAILED] when connecting to GitLab instances using self-signed certificates
Setting REQUESTS_CA_BUNDLE or adding certificates to the system CA bundle does not resolve the issue

Proposal

I was trying to understand how Runner does this when Gitlab uses a self-signed certificate (as explain in runner docs)

As I understand it:

Runner has its own internal system to deal with certificate verification
it treats regular certificates and self-signed ones differently
specifically, in the case of self-signed certificates, it makes it work by treating the self-signed certificate as if it were a trusted root CA.

Unfortunately, at this point in time, we do not have such a system in place for AIGW. We'd have to build some form of certificate management system within AIGW, similar to runner to support self-signed certificates properly.

Further details

Links / references

https://gitlab.com/gitlab-com/request-for-help/-/issues/2371