Dogfood Custom Model Set Up with GitLab Developers

Context

Our design partners are currently struggling to successfully set up all aspects of self-hosted models. Prior to going GA, and in addition to design partner feedback, we would like to solicit GitLab developers to configure self-hosted model Duo features using a GCP hosted model or a locally hosted models.

Objective

Assess Self-Hosted Model set up, focusing in the experience of setting up and configuring self-hosted models for Code Suggestions.

🙏 README

Thank you for dogfooding self-hosted models and helping us improve the experience for GitLab customers. Any and all feedback is valuable!

Follow the numbered sections below, and you should be set to use and provide feedback on your experiences.

If you have questions about the instructions below, you can comment on this issue or ask in the #g_custom_models Slack channel

To provide feedback about self-hosted model set up, please comment on this issue and include any relevant screenshots, log files, or videos.

Pre-Steps

1. Set up a self-managed Gitlab instance
2. Set up a supported LLM, either self-hosted or cloud-based
   • an example set up for Codestral with Ollama can be found here

Set up and use a self-hosted model for Code Suggestions

1. Follow the documentation here to configure your GitLab instance, AI Gateway, and model:
   • https://docs.gitlab.com/ee/administration/self_hosted_models/
   • *note that you are not required to have Gitlab Enterprise Edition in the 17.5 milestone
2. Ensure that the feature is working as expected, try out a few code completion and code generation prompts.

🗣️ Provide feedback

Your feedback is very valuable to us, especially if you have never set up a self-hosted model and configured it with Gitlab before.

Valuable areas of feedback on areas of this process may include (but are not limited to!):

• Enabling feature flags
• Setting up locally hosted AI Gateway
• Declaring a model to GL
• Configuring models to self-hosted features

Record your screen

1. Use a screen recording tool of your choice (e.g., Zoom, OBS Studio, QuickTime Player, or built-in OS tools).
2. Record your entire screen while setting up and configuring self-hosted code suggestions
3. Ensure audio is enabled to capture your verbal comments or observations.
4. Try to talk out loud about what you are seeing, thinking, and feeling.
5. Save your recording in a common format (e.g., MP4).
6. Upload your recording to your comment. If larger than 100 MB, upload to this Google Drive folder.

Out of scope

• Evaluation of the self-hosted Code Suggestions feature itself
• Problems related to setting up a self-managed Gitlab instance, or setting up the LLM itself