Skip to content

Add support for using CodeGemma for code completions in Code Suggestions

Bruno Cardoso requested to merge bc/custom-models-code-gemma-2b into master

What does this MR do and why?

This MR adds support for using a codegemma model for Code Completions (either 2b-code or 7b-code).

Reference MRs:

Part of Code Gemma 2B prompt: Code Completion (#462483 - closed)

Evaluation results

I've ran the code-suggestions eval pipeline against a local gdk instance that uses codegemma for code completions, these were the results:

model avg_similarity_score sample_size table
codegemma:2b-code 0.72 4497 dev-ai-research-0e2f8974.code_suggestion_experiments.bc_duo_chat_cs_sample_2b_full
codegemma:7b-code 0.70 5431 dev-ai-research-0e2f8974.code_suggestion_experiments.bc_duo_chat_cs_sample_7b_full
prod (code-gecko) 0.77 294 dev-ai-research-0e2f8974.code_suggestion_experiments.bc_duo_chat_cs_sample_prod_full

note: different sample sizes is due to either getting rate limited (in the code-gecko case) or some connection error. Thus I filter those out of the results.

Example query:

SELECT model, round(avg(similarity_score), 2) as avg_score, count(*) FROM `dev-ai-research-0e2f8974.code_suggestion_experiments.bc_duo_chat_cs_sample_2b_full` WHERE NOT STARTS_WITH(model_completion, '{') GROUP BY model
Example config: 
{
  "beam_config": {
    "pipeline_options": {
      "runner": "DirectRunner",
      "project": "dev-ai-research-0e2f8974",
      "region": "us-central1",
      "temp_location": "gs://prompt-library/tmp/",
      "save_main_session": false
    }
  },
  "input_bq_table": "dev-ai-research-0e2f8974.code_suggestion.input_testcases_v1",
  "output_bq_table": "dev-ai-research-0e2f8974.code_suggestion_experiments.<change_the_table_name>",
  "throttle_sec": 1,
  "batch_size": 32,
  "min_length": 0,
  "eval_setups": [
    {
      "llm_config": {
        "name": "gitlab-code-completion",
        "parameters": {
          "base_url": "http://gdk.test:3000",
          "intent": "completion"
        }
      },
      "post_transformations": [],
      "include_suffix": true
    }
  ]
}

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screen_Recording_2024-05-28_at_14.03.00

How to set up and validate locally

  1. Add a new self hosted model in http://gdk.test:3000/admin/ai/self_hosted_models as shown in the video.
  2. Set the new model to be used for Code Completions http://gdk.test:3000/admin/ai/feature_settings image
  3. Test it in web-ide. Check the Developer Tools to make sure that the completion was served by ollama/codegemma:2b.
Edited by Bruno Cardoso

Merge request reports