Add runtime field to convert mlops httpRequest.latency to a number
What
- Adds the
pubsub-latency-string-to-doublecomponent template. - The template adds a runtime field,
json.httpRequest.latency_seconds(double). A Painless script https://www.elastic.co/docs/explore-analyze/scripting/modules-scripting-painless parses it from the existing string duration field (for example,0.002406611s). - Moves
mlopsout of the main sharedstandardcomposition tier (nonprod) to be able to do this. - Adds a new
mlops_latencytier formlops. This tier composes the new component template for adding this filed with the already existingpubsub-logsdb-basecomponent.
Why
json.httpRequest.latency on pubsub-mlops-inf-gprd (and nonprod) is a duration string, not a number. You cannot query or aggregate on it. Resolves work item 4432, from INC-3390.
Nonprod only for now. Prod is a follow-up once nonprod has proved stable and functional.
Why runtime field
- It works in real Elasticsearch queries and in Watcher alerts, not only in Kibana. Elasticsearch computes the value at query time.
- It applies to old documents too. No reindex is needed.
- It is additive. The original string field stays untouched, so nothing that reads it today breaks.
Trade-off: a runtime field is not doc-values backed, so aggregations cost more per query than a real stored field would. mlops index is not high volume so this should not be an issue but can be revisitted if required. An ingest pipeline would be the better long-term if the index was high volume and if the runtime field proves too resource intensive, will try that approach.
I think we could also handle this at the pubsubbeat or vector level https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/logging/vector.md but from what I can see we don't have any examples right now of this kind of config in the vector-config at https://gitlab.com/gitlab-com/gl-infra/charts/-/tree/main/gitlab/vector-config, so this seems like possibly the most additive way to add this functionality without breaking something or having to reindex.