fix: exception when extracting vertex errors (!737) · Merge requests · GitLab.org / ModelOps / AI Assisted (formerly Applied ML) / Code Suggestions / AI Gateway

Tan Le requested to merge fix-vertex-api-error-logging into main Apr 15, 2024

What does this merge request do and why?

This MR fixes an unhandled exception caused when constructing the error messages from Vertex AI API error payload.

Traceback (most recent call last):
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
    return self.receive_nowait()
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 159, in call_next
    message = await recv_stream.receive()
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 118, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/api/middleware.py", line 110, in dispatch
    response = await call_next(request)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 165, in call_next
    raise app_exc
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 151, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/authentication.py", line 48, in __call__
    await self.app(scope, receive, send)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 191, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/api/middleware.py", line 280, in dispatch
    return await call_next(request)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 165, in call_next
    raise app_exc
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 151, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/Users/tanle/code/gitlab/ai-assist-api/.venv/lib/python3.10/site-packages/starlette/authentication.py", line 82, in async_wrapper
    return await func(*args, **kwargs)
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/api/feature_category.py", line 21, in wrapper
    return await func(*args, **kwargs)
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/api/v2/code/completions.py", line 115, in completions
    suggestion = await code_completions.execute(
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/code_suggestions/completions.py", line 57, in execute
    response = await self.engine.generate(prefix, suffix, file_name, editor_lang)
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/code_suggestions/processing/base.py", line 85, in generate
    return await self._generate(
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/code_suggestions/processing/completions.py", line 217, in _generate
    if res := await self.model.generate(
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/models/vertex_text.py", line 346, in generate
    res = await self._generate(
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/models/vertex_text.py", line 150, in _generate
    with self.instrumentator.watch():
  File "/Users/tanle/.local/share/mise/installs/python/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/instrumentators/model_requests.py", line 57, in watch
    log_exception(ex, self.labels)
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/tracking/errors.py", line 25, in log_exception
    str(ex),
  File "/Users/tanle/code/gitlab/ai-assist-api/ai_gateway/models/base.py", line 48, in __str__
    message = f"{message} {', '.join(self.details)}"
TypeError: sequence item 0: expected str instance, ErrorInfo found

Steps to reproduce

Set an incorrect Vertex project in .env.

AIGW_VERTEX_TEXT_MODEL__PROJECT=unknown-project

Start the app and send a code completion request.

curl --request POST \
  --url http://ai-gateway.gdk.test:5052/v2/completions \
  --header 'Content-Type: application/json' \
  --data '{
 "prompt_version": 1,
 "project_path": "awesome_project",
 "project_id": 23,
 "current_file": {
 	"file_name": "main.go",
 	"content_above_cursor": "func (h* HotelHandler) HandleGetHotels(c *fiber.Ctx) error {\n\thotels, err := h.Store.Hotels.GetHotels(c.Context(), nil)\n\tif err != nil {\n\t\treturn err\n\t}\n\treturn",
 	"content_below_cursor": "}"
 }
}'

Observe the above error in the log. The AI Gateway response is 500.

How to set up and validate locally

Check out to this merge request's branch.
Repeat the reproduce steps above

Confirm the model exception is correctly recorded, e.g. 403. The AI Gateway response is 200.

{
  "url": "http://ai-gateway.gdk.test:5052/v2/completions",
  "path": "/v2/completions",
  "status_code": 200,
  "method": "POST",
  "correlation_id": "172adba0ccb04f9ca4be2037b47c130d",
  "http_version": "1.1",
  "client_ip": "172.16.123.4",
  "client_port": 50395,
  "duration_s": 1.1152861249865964,
  "cpu_s": 0.03419599999999967,
  "user_agent": null,
  "gitlab_instance_id": null,
  "gitlab_global_user_id": null,
  "gitlab_host_name": null,
  "gitlab_saas_duo_pro_namespace_ids": null,
  "gitlab_saas_namespace_ids": null,
  "gitlab_realm": null,
  "meta.feature_category": "code_suggestions",
  "model_engine": "vertex-ai",
  "model_name": "code-gecko@002",
  "prompt_length": 221,
  "prompt_length_stripped": 183,
  "inference_duration_s": 1.0886952499859035,
  "suffix_length": 1,
  "prompt_symbols": {
    "comment": 1
  },
  "experiments": [
    {
      "name": "exp_truncate_suffix",
      "variant": 0
    }
  ],
  "lang": "go",
  "editor_lang": null,
  "model_exception_message": "403 Vertex Model API error: PermissionDenied Permission denied on resource project dev-ai-research-0e2f89747. [links {\n  description: \"Google developers console\"\n  url: \"https://console.developers.google.com\"\n}\n, reason: \"CONSUMER_INVALID\"\ndomain: \"googleapis.com\"\nmetadata {\n  key: \"consumer\"\n  value: \"projects/dev-ai-research-0e2f89747\"\n}\nmetadata {\n  key: \"service\"\n  value: \"aiplatform.googleapis.com\"\n}\n]",
  "model_exception_status_code": 403,
  "logger": "api.access",
  "level": "info",
  "type": "mlops",
  "stage": "main",
  "timestamp": "2024-04-16T04:16:35.195019Z",
  "message": "172.16.123.4:50395 - \"POST /v2/completions HTTP/1.1\" 200"
}

Merge request checklist

Tests added for new functionality. If not, please raise an issue to follow up.
Documentation added/updated, if needed.

Edited Apr 16, 2024 by Tan Le

fix: exception when extracting vertex errors

What does this merge request do and why?

Steps to reproduce

How to set up and validate locally

Merge request checklist

Merge request reports