Skip to content

Consolidate tokenizer logic in code suggestions

Tan Le requested to merge consolidate-tokenizers-logic into main

What does this merge request do and why?

Consolidate tokenizer logic in code suggestions.

Code completions via Vertex models utilize duplicate tokenization logic with code completions via Anthropic and code generations. This results in maintenance overhead and surprising bugs due to duplication.

This consolidates tokenization logic into TokenizerTokenStrategy and standardize the interface of CodeCompletionsLegacy. As part of this refactor, some types are also moved to processing.typing to avoid circular dependency.

How to set up and validate locally

  1. Check out to this merge request's branch.
  2. Ensure the server boots up normally. 
    poetry run ai_gateway
  3. Run a few cURL requests, esp. code completion via Vertex AI. 
       curl --request POST \
  --url http://0.0.0.0:5052/v2/completions \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
  "prompt_version": 1,
  "project_path": "awesome_project",
  "project_id": 23,
  "current_file": {
    "file_name": "main.py",
    "content_above_cursor": "def prime_numbers(n):\n\t'\'''\'''\''Generate a list of ",
    "content_below_cursor": ""
  }
}'

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Tan Le

Merge request reports