Package AI Gateway with Omnibus

added devopssystems groupdistribution group::distributionbuild labels

added sectioncore platform label

@alejandro

How do you feel about picking this one up after we finish banning?

/cc @sean_carroll

@oregand sounds good to me!

Updated the link to the Custom Models parent Epic to Group: Custom Models (&13068)

marked this issue as related to gitlab#452490 (closed)

changed the description

added For Scheduling label

added featureaddition typefeature labels

@sean_carroll is this in the scope within the %Next 1-3 releases or %Next 4-6 releases? Asking this because I want to assign a priority label.

@sean_carroll @oregand @alejandro This issue needs fleshed out, significantly, if groupdistribution is to be responsible for it.

What we know so far:

It is part of AI work
It is python
It is a "service"

What we don't know:

Where is this project source?
- Is this it?
What are the functional requirements?
- dependencies (direct, and network)
- data requirements (service, and storage)
- expected performance characteristics (cpu, memory, network, storage load)
When should it be operated (enabled)?

Where is this project source?

Is this it?

Correct!

What are the functional requirements?

Dependencies

GitLab instance (optional, AIGW_GITLAB_URL)
GitLab customers instance (optional, AIGW_CUSTOMER_PORTAL_URL)
Snowplow instance (optional, AIGW_SNOWPLOW__ENDPOINT)
Anthropic API
Google Vertex API (AIGW_VERTEX_TEXT_MODEL__ENDPOINT)

Data requirements

Working directory
Log directory

No database, cache, or persistent file storage needed

Expected performance characteristics

I'd model this after the Gitlab Webservice containers, so e.g. on k8s land the defaults for the Webservice chart should be a good start for this service, since at the moment it basically operates as an HTTP REST API server. In the future, custom models will require adapting this service to more gpu-centric workloads.

When should it be operated (enabled)?

@WarheadsSE as in, in which situations would users enable this service, or what is our timeline for having this in omnibus-gitlab? As for the former, this service should be disabled by default, as most self-managed customers will rely on cloud.gitlab.com. As for the latter, deferring to @sean_carroll.

I'll be preparing a draft MR to incorporate this service on this repo, and getting feedback from the distribution team.

Working directory

How much space is expected to "scratch" use? Does this need read/write access to the filesystem at all?

How much space is expected to "scratch" use?

@WarheadsSE sorry, what's "scratch use" in this context? If you mean how much space is required for the application files, the Dockerfile images are around 200mb of space.

Does this need read/write access to the filesystem at all?

We have code that reads from the filesystem but that seems to be unused since this MR. So unless I'm missing something, the answer seems to be "no", /cc @achueshev to double-check. Caching seems to be done all in memory as well (ref).

@alejandro What write permissions, and how much data is written into the working directory? that is zero.

What write permissions, and how much data is written into the working directory?

Ah, I see. @tle_gitlab can you weight in here? Does the AI Gateway service write stuff to the filesystem? My assumption is that it doesn't, but I don't know if tree-sitter does. What about in the future with Custom Models?

@alejandro groupcustom models does not have any plans to write to the filesystem

@oregand or @dmishunov does the AI Gateway write anything to the filesystem?

CC to @igor.drozdov

@alejandro The Hugging Face tokenizer in AI Gateway might write to the user's home folder (/root/.cache) if there are no available cache in this location, or the cache structure has changed (upstream code).

❯ dust ~/.cache/huggingface/hub/models--Salesforce--codegen2-16B
  0B       ┌── added_tokens.json                     │█                         │   0%
  0B       ├── merges.txt                            │█                         │   0%
  0B       ├── special_tokens_map.json               │█                         │   0%
  0B       ├── tokenizer.json                        │█                         │   0%
  0B       ├── tokenizer_config.json                 │█                         │   0%
  0B       ├── vocab.json                            │█                         │   0%
  0B     ┌─┴ 43b97656d3aa72b2a52449d956bd26bd83ada7e5│█                         │   0%
  0B   ┌─┴ snapshots                                 │█                         │   0%
4.0K   │ ┌── main                                    │█                         │   0%
4.0K   ├─┴ refs                                      │█                         │   0%
4.0K   │ ┌── 0204ed10c186a4c7c68f55dff8f26087a45898d6│█░░░░░░░░░░░░░░░░░░░░░░░░ │   0%
4.0K   │ ├── 31a4ebf2104713af5922ab75c18e61597a6f076e│█░░░░░░░░░░░░░░░░░░░░░░░░ │   0%
 24K   │ ├── 9b3078b1518d4ec699aa2f024a0f7f4ca6347bb0│█░░░░░░░░░░░░░░░░░░░░░░░░ │   1%
448K   │ ├── 6636bda4a1fd7a63653dffb22683b8162c8de956│████░░░░░░░░░░░░░░░░░░░░░ │  13%
780K   │ ├── 84ef7fb594b5c0979e48bdeddb60a0adef33df0b│██████░░░░░░░░░░░░░░░░░░░ │  22%
2.2M   │ ├── ce46d94afba693d05ca934d4a67fa7e219083edc│████████████████░░░░░░░░░ │  64%
3.4M   ├─┴ blobs                                     │█████████████████████████ │ 100%
3.4M ┌─┴ models--Salesforce--codegen2-16B            │█████████████████████████ │ 100%

Additionally, it seems that we also write GCP credential file if JSON_KEY env is provider (code). However, it seems like this is ONLY required in CI.

We also have ongoing work to read prompts from the filesystem.

@sean_carroll I don't think it does currently, but @shinya.maeda would be the correct person to answer that question.

Shinya, could you please help here? Does the AI Gateway write anything to the filesystem?

@tle_gitlab, out of curiosity, could you please elaborate on when we run the Hugging Face tokenizer? Is it something we always do, or should there be some conditions for that?

@dmishunov We always initialise the tokenizer on app boot.

Calculate on token usage per request.
- Code Completions:
  - Vertex AI - non-stream requests: if the third-party LLM responses do not include them (code)
- Code Generations:
  - Stream requests: always (code)
  - Non-stream requests: always (code)
Calculate remaining tokens when performing pre-processing (when adding imports, function signatures, etc.) - only applicable for Code Completions with Vertex AI models (code).

Ah! Thanks for the clarification, @tle_gitlab Makes sense.

@dmishunov @sean_carroll

@tle_gitlab is right above that some dependencies could write caches in FS. As AI Gateway is designed as stateless service today, I don't think we need PVC, but an ephemeral volume is sufficient.

Just to be clear, regarding this issue.

Several have spoken about K8s deployments, PVC, etc.

This is for implementation in Self-Managed via Omnibus GitLab, and as such, this has different though related concerns.

@sean_carroll The dependencies that @alejandro laid out above would be slightly different for the AIG dependencies required by the Custom Models team, correct? For example, we would not have the dependency of the Anthropic API or the Google Vertex API (AIGW_VERTEX_TEXT_MODEL__ENDPOINT) as we are not expecting the customers to connect to those APIs at the outset. As such, the dependencies would be:

GitLab instance (optional, AIGW_GITLAB_URL)
GitLab customers instance (optional, AIGW_CUSTOMER_PORTAL_URL)
Snowplow instance (optional, AIGW_SNOWPLOW__ENDPOINT)

@susie.bee I am not sure.

@jpcyiza can you please answer this question?

@susie.bee @sean_carroll We actually only need these env variables for the AIGW to run for self-hosted models (initially):

AIGW_GITLAB_URL=https://<your_gitlab_domain>
AIGW_GITLAB_API_URL=https://<your_gitlab_domain>/api/v4/

As the Documentation search can be handled in a local DB, and the other configurations are handled in UI in the self-hosted models admin page.

As AI Gateway is designed as stateless service today, I don't think we need PVC, but an ephemeral volume is sufficient.

@shinya.maeda does the AIGW write logs, and are they written inside the container or on the host filesystem?

and are they written inside the container or on the host filesystem?

@sean_carroll it writes logs, the location of where the logs can be written can be configured to be the host filesystem

Thank you @WarheadsSE @pursultani and apologies for the incomplete requirements. I will respond to this in the next couple of days: we are working on some groupcustom models planning at the moment.

CC @susie.bee

assigned to @alejandro

mentioned in merge request gitlab!149028 (merged)

added to epic &13393

@alejandro @oregand Just bringing it to your attention that the draft blueprint states:

Self-hosted Runway will be the preferred delivery mechanism for deploying the AI Gateway. Future options, in order of preference are:

- Runway [discussion](https://gitlab.com/gitlab-com/gl-security/security-assurance/fedramp/fedramp-certification/-/issues/452#note_1832261170)
- Kubernetes deployment [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/452490)
- Omnibus packaging [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8467

This documents that shipping the AI-gateway through omnibus is the least preferred approach, because we can't reuse it for the other GitLab environments (GET-hybrid, and isolated dedicated instances). So perhaps it is fair to hold off on this for a while if we provide the way for self-managed user to temporarily handle it themselves (gitlab#452489 (closed)).

I know it's not user friendly, and I don't want to slow anything down, but in my opinion we will need to build something to easily ship Runway-services to other environments, and I think it's preferable to do it generic so we can apply the same thing for other services.

cc @andrewn @swiskow @rnienaber

CC @sean_carroll

This documents that shipping the AI-gateway through omnibus is the least preferred approach, because we can't reuse it for the other GitLab environments (GET-hybrid, and isolated dedicated instances).

Please note that there are GitLab instances that solely rely on Omnibus Linux packages and do not use Kubernetes to manage their workload. Even GitLab reference architecture documents do not explicitly refer to Kubernetes. If this is meant for self-managed instances then we need to ship it with BOTH Omnibus and CNG.

Using the Spamcheck as example of an auxiliary service could be helpful:

Omnibus GitLab: #6259 (closed)
GitLab Chart: gitlab-org/charts/gitlab#2936 (closed)

@alejandro @reprazent @sean_carroll @oregand @swiskow @rnienaber: the effort in doing this would be significant, and would be once-off for the AI gateway. It would be better, IMO, to put this effort into supporting Runway-on-Kubernetes for Self-Managed use-cases.

That approach would bring significant advantages over this approach

For application developers: a single model for AI Gateway deployment across cloud and SAAS
For platform teams: a single set of best practices that can be followed by self-managed and cloud deployments.
For GitLab: a single effort to deploy many Runway services, not just a single service.

cc @fzimmer @marin

@andrewn , happy to help if you need any changes on the AIGW side to implement this idea.

@andrewn

This sounds like a great approach, and I am generally a fan of leaning into Kubernetes, @alejandro what do you think? If we're aligned this would be a better approach we can repurpose this issue.

shipping the AI-gateway through omnibus is the least preferred approach

@reprazent yes this is correct

groupcustom models is agnostic about how the AI Gateway is deployed and are happy to follow any wider company direction. For the MVP we are moving forward with a Docker install, but as noted on the Blueprint this is a temporary measure.

put this effort into supporting Runway-on-Kubernetes for Self-Managed use-cases.

@andrewn @oregand @alejandro if the effort to implement in Omnibus is low (as Python is already packaged), are there any reasons not to proceed with it?

the effort in doing this would be significant, and would be once-off for the AI gateway. It would be better, IMO, to put this effort into supporting Runway-on-Kubernetes

if the effort to implement in Omnibus is low (as Python is already packaged), are there any reasons not to proceed with it?

This is the part I'm not clear on yet. I considered the scope of this issue to be basically following the Adding a new Service to Omnibus doc, which should be a single MR effort; so it seemed to me that if we were able to quickly ship that we'd unlock at least the customers interested in self-managed AI Gateway that are already running omnibus. Maybe I'm missing some necessary follow-up steps to that?

@alejandro

Thats a interesting point, perhaps I had misunderstood too.

If I considered the scope of this issue to be basically following the Adding a new Service to Omnibus doc, is the scope entirely and its a single MR, I see no reason not to do it for unlocking customers already running omnibus.

@andrewn

Are we misaligned on the scope of the issue or missing some steps perhaps?

if the effort to implement in Omnibus is low (as Python is already packaged), are there any reasons not to proceed with it?

What about all the upstream C/C++ dependencies and libraries that need to be built and packaged in order to support the Python libraries included in the AI Gateway? have you included the packaging and support for those in this assessment? https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/poetry.lock. As I understand it, some of these libraries are pretty complicated to build, and lend themselves much better to being packaged in a container than in Omnibus.

Secondly, once this arrives in Omnibus, it's going to stay there, putting a huge burden on the team going forward. Providing an early-adopter client with a temporary approach as discussed previously gives the teams the space to figure out how to do this properly. To put it bluntly: jamming this into Omnibus feels like a missed opportunity to move certain auxiliary workloads in Kubernetes which will ultimately - in my opinion - be better for us to manage and our customers to operate.

@oregand @alejandro @sean_carroll

What about all the upstream C/C++ dependencies and libraries that need to be built and packaged in order to support the Python libraries included in the AI Gateway?

Aha, I had not properly weighted this. There's a chance we're lucky and the current gcc/libs bundle that omnibus already relies on is sufficient, which is what I was tacitly assuming, but there's also a chance that it can get quite messy.

Providing an early-adopter client with a temporary approach as discussed previously gives the teams the space to figure out how to do this properly. To put it bluntly: jamming this into Omnibus feels like a missed opportunity to move certain auxiliary workloads in Kubernetes

@andrewn I guess what I'm not quite grasping is whether we'll be pushing Runway-on-Kubernetes as the preferred path for all customers. It will definitely be the preferred path for us (GitLab.com, Dedicated), but my approach to this issue was "meet customers where they're at", assuming that customers running containerized deployments would be unlikely to deploy via Omnibus, and vice-versa, and thus implementing only one of the options would be mostly meaningless for the other segment of customers. I could be wrong in this assumption (see this comment for an opposing perspective). Specifically for the purpose of self-managed, my understanding is that one of the major customers interested is running an omnibus-based HA deployment, so my push for this issue is based on two assumptions:

This would require little effort and thus not detract from our focus on Runway-on-Kubernetes. This could be hastily proven wrong by dependency/library management. Maybe whipping out a quick POC would help us establish this
This would be a major win with a large customer that's already running on omnibus, as it would make them adopt SM AIGW quickly. Maybe here it'd be helpful reaching out to the customers and validating if adding a containerized deployment to their setup would be an acceptable path forward/temporary solution.

Do we have a good idea of "where customers are at" ? I understand that Omnibus is a popular installation method for GitLab.

However, have we verified that customers who run Omnibus GitLab are unwilling or unable to add an AI Gateway deployment to a Kubernetes cluster they already manage within their business IT infrastructure?

When we do the analysis and talk to customers, we may find that a relatively small number of them cannot commit to a Kubernetes deployment of the AI Gateway alongside the Omnibus deployment. We may also find the contrary, but I think it merits investigation before we commit to something that is closer to a one way than a two way door.

groupcustom models is agnostic to how AI Gateway is deployed for self-managed.

We do have 2 customers interested in the feature who are on HA Omnibus (if that is the right term), but we will start with the Docker deployment Document installing AI Gateway via Docker (gitlab#452489 - closed) as a stop-gap. A longer term deployment for these large customers would depend on their specific needs and where we are in terms of the other options.

The customers are running K8s elsewhere in their infrastructure, so Helm Charts for AI Gateway deployment (gitlab#452490 - closed) would be a possibility until Runway for self-managed is ready.

If a decision is made to not proceed with the Omnibus option, please let me know and I'll update the blueprint.

CC @andrewn @oregand @alejandro @reprazent

See also Deployment of the AI Gateway using Runway (gitlab#455854)

The customers are running K8s elsewhere in their infrastructure, so Helm Charts for AI Gateway deployment (gitlab#452490 - closed) would be a possibility until Runway for self-managed is ready ... not proceed with the Omnibus option

@sean_carroll we ended up recommending the same approach for any future on-premise deployment of GitLab Opstrace. This is not unprecedented.

we ended up recommending the same approach for any future on-premise deployment of GitLab Opstrace. This is not unprecedented.

Happy to go with our established precedent then. I'm going to label this as workflowproblem validation and remove the For Scheduling, to reflect the fact that we're not going to prioritize this issue. Thanks everyone for the discussion! /cc @oregand @pursultani @sean_carroll

Thank you @alejandro

@sean_carroll

I will defer to you here on how you want to proceed for this one specifically, it looks to me like we would rather peruse Helm Charts for AI Gateway deployment (gitlab#452490 - closed) • Patrick Cyiza • 17.3 which is already picked up and in flight

mentioned in issue gitlab-com/gl-infra/scalability#3395

mentioned in issue gitlab#452490 (closed)

marked this issue as related to gitlab#455854

Tagging @sissiyao and @balasankarc given the potential impact of this on Omnibus GitLab and group::distributionbuild.

From distribution perspective, there are two major issues:

Software definitions, i.e. build instructions, for the required dependencies of AIGW. For example the extensions of transitive dependencies such libtorch and/or libtensorflow.
Package size and the excess from AIGW.

Thank you for including us in the loop, @pursultani .

@balasankarc , can you take a look at this requirement?

/cc @dorrino

added workflowproblem validation label and removed For Scheduling label

unassigned @alejandro

Based on this thread we can close this issue without any action required.

This is the long-term direction Deployment of the AI Gateway using Runway (gitlab#455854)

CC @pursultani @sissiyao @andrewn @alejandro

closed

marked this issue as related to gitlab#463773 (closed)

mentioned in issue gitlab#463773 (closed)

mentioned in issue gitlab#455854

reopened

added Category:Build label

Reopening this issue for further discussion, we have some high-profile customers who have been struggling to get the Docker install of AI Gateway running. More context in the Internal Note below.