Skip to content

Fix the v3 code generations dependency injection

Andras Herczeg requested to merge ah/fix-v3-generations-dependency into main

What does this merge request do and why?

A bug was introduced to the v3 code generation endpoint when we allowed the dynamic selection of anthropic models. This MR fixes this problem with the smallest possible footprint; another MR will follow to introduce the model selection to the v3 endpoint as well.

Steps to reproduce

  1. Check out the main branch

  2. Send a cURL request to the /v3/code/completions endpoint

    curl --request POST \
      --url http://codesuggestions.gdk.test:5999/v3/code/completions \
      --header 'Content-Type: application/json' \
      --header 'X-Gitlab-Authentication-Type: oidc' \
      --data '{
        "prompt_components": [
            {
                "type": "code_editor_generation",
                "payload": {
                    "file_name": "test",
                    "content_above_cursor": "func hello_world(){",
                    "content_below_cursor": "",
                    "model_provider": "anthropic",
                    "language_identifier": "go",
                    "prompt": "Human: Write a golang function that prints hello world."
                },
                "metadata": {
                    "source": "Gitlab EE",
                    "version": "16.3"
                }
            }
        ]
    }'
  3. Observe 500 HTTP response and error in the log

    {
      "url": "http://codesuggestions.gdk.test:5999/v3/code/completions",
      "path": "/v3/code/completions",
      "status_code": 500,
      "method": "POST",
      "correlation_id": "bc228d57d2bb49a4b59ebdc0d3f1af83",
      "http_version": "1.1",
      "client_ip": "172.17.0.1",
      "client_port": 47568,
      "duration_s": 0.061061542997776996,
      "cpu_s": 0.038238288999998815,
      "user_agent": null,
      "gitlab_instance_id": null,
      "gitlab_global_user_id": null,
      "gitlab_host_name": null,
      "gitlab_saas_namespace_ids": null,
      "gitlab_realm": null,
      "meta.feature_category": "code_suggestions",
      "exception": {
        "message": "CodeGenerations.__init__() missing 1 required positional argument: 'model'",
        "backtrace": "Traceback (most recent call last):\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/anyio/streams/memory.py\", line 98, in receive\n    return self.receive_nowait()\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/anyio/streams/memory.py\", line 93, in receive_nowait\n    raise WouldBlock\nanyio.WouldBlock\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 78, in call_next\n    message = await recv_stream.receive()\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/anyio/streams/memory.py\", line 118, in receive\n    raise EndOfStream\nanyio.EndOfStream\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/app/ai_gateway/api/middleware.py\", line 110, in dispatch\n    response = await call_next(request)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 84, in call_next\n    raise app_exc\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 70, in coro\n    await self.app(scope, receive_or_disconnect, send_no_error)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/authentication.py\", line 48, in __call__\n    await self.app(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 108, in __call__\n    response = await self.dispatch_func(request, call_next)\n  File \"/app/ai_gateway/api/middleware.py\", line 287, in dispatch\n    return await call_next(request)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 84, in call_next\n    raise app_exc\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 70, in coro\n    await self.app(scope, receive_or_disconnect, send_no_error)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py\", line 79, in __call__\n    raise exc\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py\", line 68, in __call__\n    await self.app(scope, receive, sender)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py\", line 20, in __call__\n    raise e\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py\", line 17, in __call__\n    await self.app(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/routing.py\", line 718, in __call__\n    await route.handle(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/routing.py\", line 276, in handle\n    await self.app(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/routing.py\", line 66, in app\n    response = await func(request)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/routing.py\", line 274, in app\n    raw_response = await run_endpoint_function(\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/routing.py\", line 191, in run_endpoint_function\n    return await dependant.call(**values)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/authentication.py\", line 76, in async_wrapper\n    return await func(*args, **kwargs)\n  File \"/app/ai_gateway/api/feature_category.py\", line 21, in wrapper\n    return await func(*args, **kwargs)\n  File \"/app/ai_gateway/api/v3/code/completions.py\", line 50, in completions\n    return await code_generation(payload=component.payload)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/dependency_injector/wiring.py\", line 994, in _patched\n    return await _async_inject(\n  File \"src/dependency_injector/_cwiring.pyx\", line 66, in _async_inject\n  File \"/app/ai_gateway/api/v3/code/completions.py\", line 103, in code_generation\n    engine = code_generations_anthropic()\n  File \"src/dependency_injector/providers.pyx\", line 225, in dependency_injector.providers.Provider.__call__\n  File \"src/dependency_injector/providers.pyx\", line 2689, in dependency_injector.providers.Factory._provide\n  File \"src/dependency_injector/providers.pxd\", line 650, in dependency_injector.providers.__factory_call\n  File \"src/dependency_injector/providers.pxd\", line 608, in dependency_injector.providers.__call\nTypeError: CodeGenerations.__init__() missing 1 required positional argument: 'model'\n"
      },
      "logger": "api.access",
      "level": "info",
      "type": "mlops",
      "stage": "main",
      "timestamp": "2024-01-04T00:10:16.539103Z",
      "message": "172.17.0.1:47568 - \"POST /v3/code/completions HTTP/1.1\" 500"
    }

How to set up and validate locally

  1. Check out to this merge request's branch.
  2. Ensure a local Docker image built successfully.
    docker buildx build --platform linux/amd64 \
      -t ai-gateway:dev .
  3. Run a local service on Docker.
    docker run --platform linux/amd64 --rm \
      -p 5052:5052 \
      -e AUTH_BYPASS_EXTERNAL=true \
      -v $PWD:/app -it ai-gateway:dev
  4. Run the following cURL request and confirm 200 HTTP response and no error in log
Code Generations - Anthropic - Default
curl --request POST \
  --url http://codesuggestions.gdk.test:5999/v3/code/completions \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
    "prompt_components": [
        {
            "type": "code_editor_generation",
            "payload": {
                "file_name": "test",
                "content_above_cursor": "func hello_world(){",
                "content_below_cursor": "",
                "model_provider": "anthropic",
                "language_identifier": "go",
                "prompt": "Human: Write a golang function that prints hello world."
            },
            "metadata": {
                "source": "Gitlab EE",
                "version": "16.3"
            }
        }
    ]
}'

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Blocked by !530 (merged)

Edited by Tan Le

Merge request reports