Skip to content

Add VertexAI cost tracking metric

Matthias Käppler requested to merge 205-track-vertex-cost into main

We need to track data that feeds into 3P cloud provider costs for AI services. We are already doing this for OpenAI but it was never added for VertexAI. See also https://gitlab.com/gitlab-org/gitlab/-/issues/419238. We currently use the gitlab_cloud_cost_spend_entry_total Prometheus counter for this.

In this MR I extend this counter to also track different model inputs and output for VertexAI calls. Google charges us by model input and output characters, so we track with characters as the unit here, not tokens. I also added a model label since cost can be different per model so we cannot lump these data together.

Sample call:

curl -v -H'content-type: application/json' -d'{
    "prompt_version": 1,
    "current_file": {
      "file_name": "test.py",
      "content_above_cursor": "def is_even(n: int) ->",
      "content_below_cursor": "end"
    }
  }' localhost:5001/v2/completions | jq

I currently count all prompt characters (prefix + suffix) minus whitespace as model "input", and the text property of the model response minus whitespace as "output". Google isn't very specific about what actually counts as the input or prompt here:

Generative AI support on Vertex AI charges by every 1,000 characters of input (prompt) and every 1,000 characters of output (response). Characters are counted by UTF-8 code points and white space is excluded from the count.

Some models seems to ignore the suffix so I count this as 0 (length of blank string). Gecko seems to use both prefix and suffix. However, I reached out to our GCP CE and he confirmed that:

All of the Prefix + Suffix Characters to be considered as input Prompt. and hence are counted toward your cost.

The resulting metric looks like this:

curl -s localhost:8082/metrics | grep gitlab_cloud_cost_spend_entry_total
grep: warning: GREP_COLOR='1;33' is deprecated; use GREP_COLORS='mt=1;33'
# HELP gitlab_cloud_cost_spend_entry_total Number of units spent per vendor entry
# TYPE gitlab_cloud_cost_spend_entry_total counter
gitlab_cloud_cost_spend_entry_total{item="completions/completion/input",model="code-gecko",unit="characters",vendor="vertex-ai"} 439.0
gitlab_cloud_cost_spend_entry_total{item="completions/completion/output",model="code-gecko",unit="characters",vendor="vertex-ai"} 234.0

Refs #205

Edited by Matthias Käppler

Merge request reports