Map duration_s log fields as float in nonprod
What
Map json.duration_s and json.*_duration_s log fields as float in nonprod. This affects the Rails and Sidekiq indices. Prod follows in a separate MR once nonprod has run stable, as with !226 (merged).
The template is composed into the high_rails tier (rails) and the high tier (sidekiq, plus gitaly and workhorse, which share that tier). Gitaly and Workhorse currently log no *_duration_s fields, so nothing changes for them; the template only adds a rule for future fields with matching names.
Why
Most Rails per-database duration fields are stored as whole seconds, so sub-second database and transaction times can't be analysed from logs. In today's prod Rails write index (.ds-pubsub-rails-inf-gprd-2026.10.19-002041), json.db_main_duration_s is long, and across all 128 Rails backing indices it is long in 127.
Across prod indices, 31 of Rails' 160 *_duration_s fields and 25 of Sidekiq's 152 are long in at least one backing index.
In that index, of 322,354,015 documents with database time, json.db_duration_s (mapped float) has a fractional part in 322,354,009, while json.db_main_duration_s is above 0 in only 3,459 and has a fractional part in none.
Cause
- The app writes decimals:
lib/gitlab/metrics/subscribers/active_record.rblogs each duration with.to_f.round(3), so a request with no main-primary time logs0.0. - pubsubbeat decodes each message into Go
float64(gitlab-org/pubsubbeat,beater/pubsubbeat.go) and libbeat re-encodes it withstrconv.AppendFloat(f, 'g', -1)(go-structform v0.0.7,json/visitor.go), which writes0.0as0and2.0as2. - The Rails indices use
index.mode: logsdb(pubsub-logsdb-base) with no explicit mapping for these fields. Dynamic mapping fixes each field's type when a new backing index first sees it:longfor a whole number,floatotherwise. - So fields that are usually exactly 0, like
db_main_duration_s, usually map aslong. Fields that are almost never 0, likedb_duration_s(Rails' total, only logged when a query ran), map asfloat. - After that, decimals are truncated on indexing. logsdb's synthetic
_sourceis rebuilt from the indexed values, so the stored document shows the truncated value too. - Found while investigating INC-15305 (gitlab-com/gl-infra/production work item 23261). Related Rails change: gitlab-org/gitlab!263117 adds
db_*_held_duration_sand related fields, which need decimals to be useful.
How to validate the problem yourself
Discover (no special access), on https://log.gprd.gitlab.net:
- Open Discover, choose the Rails data view (
pubsub-rails-inf-gprd-*), and set the time range to the last 15 minutes. - Query:
json.db_duration_s >= 0.3 and json.db_main_count >= 1 - Add the columns
json.db_duration_s,json.db_main_duration_s,json.db_main_replica_duration_sandjson.db_ci_duration_s. db_duration_sis the total database time and the other columns are its parts. The total has decimals, while the parts read0.
Dev Tools (exact numbers)
- Field types across indices:
GET pubsub-rails-inf-gprd/_field_caps?fields=json.db_main_duration_s,json.db_main_txn_duration_s,json.db_duration_s,json.duration_s - The write index is the last entry in
indicesofGET _data_stream/pubsub-rails-inf-gprd. Its mapping and mode:ExpectGET <write-index>/_mapping/field/json.db_main_duration_s,json.db_duration_s GET <write-index>/_settings/index.mode,index.mapping.source.mode?include_defaults=true&flat_settings=truelongvsfloat, andlogsdbwithSYNTHETICsource. - Whole vs fractional values in the write index (scans a day of data, so add an
@timestamprange to make it faster):GET <write-index>/_search { "size": 0, "track_total_hits": true, "query": { "range": { "json.db_duration_s": { "gt": 0 } } }, "aggs": { "total_has_decimals": { "filter": { "script": { "script": "def v = doc['json.db_duration_s']; return v.size() > 0 && v.value != Math.floor(v.value)" } } }, "main_nonzero": { "filter": { "range": { "json.db_main_duration_s": { "gt": 0 } } } }, "main_has_decimals": { "filter": { "script": { "script": "def v = doc['json.db_main_duration_s']; return v.size() > 0 && v.value != Math.floor(v.value)" } } } } }
References
- Plan and tracking: https://gitlab.com/gitlab-org/gitlab/-/work_items/634464
- gitlab-org/gitlab!263117 (new held-time log fields that need decimals)
Change
- New component template
pubsub-duration-seconds-as-floatwith one dynamic template. It matches only numeric values (long,double), so strings and objects with matching names are unaffected. - Composed into
high_railsandhighfor both nonprod groups (pubsub,pubsub_stgsub_ref). Gitaly and Workhorse are inhighbut have no matching fields today.
Impact
- Takes effect from the next rollover of each data stream. Existing backing indices keep their current mapping.
- During retention, data views see
longin older indices andfloatin newer ones. Both are numeric, so queries and aggregations keep working; Kibana may show a type conflict for these fields until old indices age out. float(4 bytes) uses less storage thanlong(8 bytes). It keeps about 7 significant digits: millisecond precision up to about 2 hours.- Why
float, notdouble: most of these fields are alreadyfloatin most backing indices. A field with two number types across indices can't be sorted on a time range that spans both, sofloatkeeps the most fields sortable during retention (thanks to @jverdoni for the field counts).
Validation after merge
- After rollover,
GET pubsub-rails-inf-gstg/_field_caps?fields=json.db_main_duration_sshowsfloatfor the new backing index. - The Discover steps above, against the
gstgdata view on https://nonprod-log.gitlab.net, show decimals injson.db_main_duration_s.
