Skip to content

Shards failed error on some Kibana queries

Some Kibana searches are failing with X of Y Shards Failed when doing generic queries (i.e., searching for "namespace/example" with no fields selected. The Message for each error is failed to create query: field expansion matches too many fields, limit: 1024, got: 1033 or something similar. I saw this on pubsub-sidekiq-inf-gprd*, pubsub-rails-inf-gprd*,

Here are two examples that trigger the error: https://log.gprd.gitlab.net/goto/20d84cc0-8adb-11ec-a649-b7cbb8e4f62e https://log.gprd.gitlab.net/goto/4636a520-8adb-11ec-9dd2-93d354bef8e7

Full error text:

query_shard_exception at shard 0index pubsub-rails-inf-gprd-008276node X_S81FAQTL-fNlSqlsFHcQ

Type
    query_shard_exception
Reason
    failed to create query: field expansion matches too many fields, limit: 1024, got: 1033
Index uuid
    Z3RKVJtxSCScf9AsEME7tg
Index
    pubsub-rails-inf-gprd-008276
Caused by type
    illegal_argument_exception
Caused by reason
    field expansion matches too many fields, limit: 1024, got: 1033