Map duration_s log fields as float in nonprod

What

Map json.duration_s and json.*_duration_s log fields as float in nonprod. This affects the Rails and Sidekiq indices. Prod follows in a separate MR once nonprod has run stable, as with !226 (merged).

The template is composed into the high_rails tier (rails) and the high tier (sidekiq, plus gitaly and workhorse, which share that tier). Gitaly and Workhorse currently log no *_duration_s fields, so nothing changes for them; the template only adds a rule for future fields with matching names.

Why

Most Rails per-database duration fields are stored as whole seconds, so sub-second database and transaction times can't be analysed from logs. In today's prod Rails write index (.ds-pubsub-rails-inf-gprd-2026.10.19-002041), json.db_main_duration_s is long, and across all 128 Rails backing indices it is long in 127.

Across prod indices, 31 of Rails' 160 *_duration_s fields and 25 of Sidekiq's 152 are long in at least one backing index.

In that index, of 322,354,015 documents with database time, json.db_duration_s (mapped float) has a fractional part in 322,354,009, while json.db_main_duration_s is above 0 in only 3,459 and has a fractional part in none.

Screenshot_2026-10-19_at_3.04.41_pm

Cause
  • The app writes decimals: lib/gitlab/metrics/subscribers/active_record.rb logs each duration with .to_f.round(3), so a request with no main-primary time logs 0.0.
  • pubsubbeat decodes each message into Go float64 (gitlab-org/pubsubbeat, beater/pubsubbeat.go) and libbeat re-encodes it with strconv.AppendFloat(f, 'g', -1) (go-structform v0.0.7, json/visitor.go), which writes 0.0 as 0 and 2.0 as 2.
  • The Rails indices use index.mode: logsdb (pubsub-logsdb-base) with no explicit mapping for these fields. Dynamic mapping fixes each field's type when a new backing index first sees it: long for a whole number, float otherwise.
  • So fields that are usually exactly 0, like db_main_duration_s, usually map as long. Fields that are almost never 0, like db_duration_s (Rails' total, only logged when a query ran), map as float.
  • After that, decimals are truncated on indexing. logsdb's synthetic _source is rebuilt from the indexed values, so the stored document shows the truncated value too.
  • Found while investigating INC-15305 (gitlab-com/gl-infra/production work item 23261). Related Rails change: gitlab-org/gitlab!263117 adds db_*_held_duration_s and related fields, which need decimals to be useful.

How to validate the problem yourself

Discover (no special access), on https://log.gprd.gitlab.net:

  1. Open Discover, choose the Rails data view (pubsub-rails-inf-gprd-*), and set the time range to the last 15 minutes.
  2. Query: json.db_duration_s >= 0.3 and json.db_main_count >= 1
  3. Add the columns json.db_duration_s, json.db_main_duration_s, json.db_main_replica_duration_s and json.db_ci_duration_s.
  4. db_duration_s is the total database time and the other columns are its parts. The total has decimals, while the parts read 0.
Dev Tools (exact numbers)
  1. Field types across indices:
    GET pubsub-rails-inf-gprd/_field_caps?fields=json.db_main_duration_s,json.db_main_txn_duration_s,json.db_duration_s,json.duration_s
  2. The write index is the last entry in indices of GET _data_stream/pubsub-rails-inf-gprd. Its mapping and mode:
    GET <write-index>/_mapping/field/json.db_main_duration_s,json.db_duration_s
    GET <write-index>/_settings/index.mode,index.mapping.source.mode?include_defaults=true&flat_settings=true
    Expect long vs float, and logsdb with SYNTHETIC source.
  3. Whole vs fractional values in the write index (scans a day of data, so add an @timestamp range to make it faster):
    GET <write-index>/_search
    {
      "size": 0,
      "track_total_hits": true,
      "query": { "range": { "json.db_duration_s": { "gt": 0 } } },
      "aggs": {
        "total_has_decimals": { "filter": { "script": { "script": "def v = doc['json.db_duration_s']; return v.size() > 0 && v.value != Math.floor(v.value)" } } },
        "main_nonzero":       { "filter": { "range": { "json.db_main_duration_s": { "gt": 0 } } } },
        "main_has_decimals":  { "filter": { "script": { "script": "def v = doc['json.db_main_duration_s']; return v.size() > 0 && v.value != Math.floor(v.value)" } } }
      }
    }

References

Change

  • New component template pubsub-duration-seconds-as-float with one dynamic template. It matches only numeric values (long, double), so strings and objects with matching names are unaffected.
  • Composed into high_rails and high for both nonprod groups (pubsub, pubsub_stgsub_ref). Gitaly and Workhorse are in high but have no matching fields today.

Impact

  • Takes effect from the next rollover of each data stream. Existing backing indices keep their current mapping.
  • During retention, data views see long in older indices and float in newer ones. Both are numeric, so queries and aggregations keep working; Kibana may show a type conflict for these fields until old indices age out.
  • float (4 bytes) uses less storage than long (8 bytes). It keeps about 7 significant digits: millisecond precision up to about 2 hours.
  • Why float, not double: most of these fields are already float in most backing indices. A field with two number types across indices can't be sorted on a time range that spans both, so float keeps the most fields sortable during retention (thanks to @jverdoni for the field counts).

Validation after merge

  1. After rollover, GET pubsub-rails-inf-gstg/_field_caps?fields=json.db_main_duration_s shows float for the new backing index.
  2. The Discover steps above, against the gstg data view on https://nonprod-log.gitlab.net, show decimals in json.db_main_duration_s.
Edited by Alex Pooley

Merge request reports

Loading
Loading