Fluentd - "429 Too Many Requests Ingestion rate limit exceeded" when shipping logs to loki
Summary
I've noticed during deployments that after a random period of time (which may coincide with loki-gateway not being available and fluentd having a bigger log chunk to send) fluentd fails to flush the buffers and send the logs to loki:
2025-06-11 13:39:44 +0000 [warn]: #0 [clusterflow:cattle-logging-system:all-logs:clusteroutput:cattle-logging-system:loki] failed to flush the buffer. retry_times=10 next_retry_time=2025-06-11 13:58:00 +0000 chunk="6374b88226f677d0636fd544da71c68a" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user management-cluster (limit: 2097152 bytes/sec) while attempting to ingest '4624' lines totaling '6570112' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
2025-06-11 13:39:44 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-grafana-loki-1.2.20/lib/fluent/plugin/out_loki.rb:171:in `write'
2025-06-11 13:39:44 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1225:in `try_flush'
2025-06-11 13:39:44 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1540:in `flush_thread_run'
2025-06-11 13:39:44 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
2025-06-11 13:39:44 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2025-06-11 13:58:00 +0000 [warn]: #0 [clusterflow:cattle-logging-system:all-logs:clusteroutput:cattle-logging-system:loki] failed to write post to http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user management-cluster (limit: 2097152 bytes/sec) while attempting to ingest '4624' lines totaling '6570112' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)loki-gateway logs:
100.72.186.166 - management-cluster [11/Jun/2025:13:21:21 +0000] 204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "Ruby" "-"
100.72.186.166 - management-cluster [11/Jun/2025:13:21:44 +0000] 429 "POST /loki/api/v1/push HTTP/1.1" 241 "-" "Ruby" "-"
100.72.186.166 - management-cluster [11/Jun/2025:13:22:49 +0000] 429 "POST /loki/api/v1/push HTTP/1.1" 241 "-" "Ruby" "-"
100.72.186.166 - management-cluster [11/Jun/2025:13:30:33 +0000] 429 "POST /loki/api/v1/push HTTP/1.1" 241 "-" "Ruby" "-"
100.72.186.166 - management-cluster [11/Jun/2025:13:39:44 +0000] 429 "POST /loki/api/v1/push HTTP/1.1" 241 "-" "Ruby" "-"
100.72.186.166 - management-cluster [11/Jun/2025:13:58:00 +0000] 429 "POST /loki/api/v1/push HTTP/1.1" 241 "-" "Ruby" "-"