Skip to content

add json.params as text field to index template

Igor requested to merge index-template-json-params into master

Sample failure:

2020-02-20_10:56:58.63721 2020-02-20T10:56:58.637Z	WARN	elasticsearch/client.go:511	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf8bb75e6a99e0e9, ext:135481088183158, loc:(*time.Location)(0x3dcf440)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"079e12a1-60c2-457c-a093-5d93f3032d0f", "hostname":"pubsub-duplicate-rails-inf-gprd", "id":"153e9ea8-766d-47eb-be71-2bdb0ae79941", "type":"pubsubbeat", "version":"7.5.1"}, "ecs":common.MapStr{"version":"1.4.0"}, "host":common.MapStr{"name":"pubsub-duplicate-rails-inf-gprd"}, "json":common.MapStr{"correlation_id":"tH6t2EBUqN4", "db":19.51, "duration":65.29, "environment":"gprd", "fqdn":"api-26-sv-gprd.c.gitlab-production.internal", "host":"gitlab.com", "hostname":"api-26-sv-gprd", "method":"POST", "params":[]interface {}{common.MapStr{"key":"private_token", "value":"[FILTERED]"}, "..."}, "path":"/api/v4/projects/13665317/mirror/pull", "queue_duration":9.96, "remote_ip":"18.234.32.225, 10.216.1.41, 18.234.32.225", "route":"/api/:version/projects/:id/mirror/pull", "severity":"INFO", "shard":"default", "stage":"main", "status":200, "tag":"rails.api", "tier":"sv", "time":"2020-02-20T10:56:41.758Z", "type":"api", "ua":"Bitbucket-Webhooks/2.0", "user_id":4.310847e+06, "username":"simonlindgren", "view":45.78}, "message":"{\"time\":\"2020-02-20T10:56:41.758Z\",\"severity\":\"INFO\",\"duration\":65.29,\"db\":19.51,\"view\":45.78,\"status\":200,\"method\":\"POST\",\"path\":\"/api/v4/projects/13665317/mirror/pull\",\"params\":[{\"key\":\"private_token\",\"value\":\"[FILTERED]\"},\"...\"],\"host\":\"gitlab.com\",\"remote_ip\":\"18.234.32.225, 10.216.1.41, 18.234.32.225\",\"ua\":\"Bitbucket-Webhooks/2.0\",\"route\":\"/api/:version/projects/:id/mirror/pull\",\"user_id\":4310847,\"username\":\"simonlindgren\",\"queue_duration\":9.96,\"correlation_id\":\"tH6t2EBUqN4\",\"tag\":\"rails.api\",\"environment\":\"gprd\",\"hostname\":\"api-26-sv-gprd\",\"fqdn\":\"api-26-sv-gprd.c.gitlab-production.internal\",\"stage\":\"main\",\"shard\":\"default\",\"tier\":\"sv\",\"type\":\"api\",\"message\":null}", "message_id":"1009103051006705", "publish_time":common.Time{wall:0x27e02140, ext:63717793017, loc:(*time.Location)(nil)}, "type":"pubsub-duplicate-rails-inf-gprd"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [json.params] tried to parse field [null] as object, but found a concrete value"}

Current inferred mapping:

GET pubsub-rails-inf-gprd-001269

...
            "params" : {
              "properties" : {
                "key" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "value" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
...

While I'm not a huge fan of maintaining these text exceptions, this one is probably going to make our lives easier.

refs https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8931

Merge request reports