Skip to content

Support prompt template with the prepended language identifier

Alexander Chueshev requested to merge add-lang-to-prompt into main

This MR updates the prompt template by adding a language identifier to it. To get the language identifier, we follow the same approach that was used in preprocessing the data to fine-tune the codegen model in https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/model-dev/-/blob/main/data/df/preprocessing.py#L233.

Briefly, we infer the language id from the file name extension. Examples:

  • f.py -> <python>
  • f.min.js -> <js>

We do not prepend a language identifier if the filename extension was not recognized and is not in the predefined list.

How to run the server locally

  1. Create virtualenv and init shell: poetry shell
  2. Install dependencies: poetry install
  3. Update the .env file in the root folder with the following variables:
    AUTH_BYPASS_EXTERNAL=true
    TRITON_HOST=localhost
    TRITON_PORT=8080
    FASTAPI_DOCS_URL=/docs
    FASTAPI_OPENAPI_URL=/openapi.json
    FASTAPI_API_PORT=5052
  4. Get k8s credentials to access our k8s cluster: gcloud container clusters get-credentials ai-assist --zone us-central1-c --project unreview-poc-390200e5
  5. Port-forward the triton server to access it locally: kubectl port-forward svc/model-k8s-triton -n fauxpilot 8080:8080 --address='0.0.0.0'
  6. Start the model-gateway server locally: poetry run codesuggestions
  7. Open http://0.0.0.0:5052/docs in your browser and run any requests to the codegen model

Ref: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/22BLO

Edited by Alexander Chueshev

Merge request reports