Shards failed error on some Kibana queries
Some Kibana searches are failing with X of Y Shards Failed when doing generic queries (i.e., searching for "namespace/example" with no fields selected. The Message for each error is failed to create query: field expansion matches too many fields, limit: 1024, got: 1033 or something similar. I saw this on pubsub-sidekiq-inf-gprd*, pubsub-rails-inf-gprd*,
Here are two examples that trigger the error: https://log.gprd.gitlab.net/goto/20d84cc0-8adb-11ec-a649-b7cbb8e4f62e https://log.gprd.gitlab.net/goto/4636a520-8adb-11ec-9dd2-93d354bef8e7
Full error text:
query_shard_exception at shard 0index pubsub-rails-inf-gprd-008276node X_S81FAQTL-fNlSqlsFHcQ
Type
query_shard_exception
Reason
failed to create query: field expansion matches too many fields, limit: 1024, got: 1033
Index uuid
Z3RKVJtxSCScf9AsEME7tg
Index
pubsub-rails-inf-gprd-008276
Caused by type
illegal_argument_exception
Caused by reason
field expansion matches too many fields, limit: 1024, got: 1033