Add option to suppress "Waiting for pod to be running" messages in Kubernetes executor job logs

Problem to Solve

When using the Kubernetes executor, the runner outputs "Waiting for pod to be running" messages to the job log while waiting for a new build pod to be scheduled. When there are substantial delays in pod scheduling (due to resource constraints, cluster autoscaling, or other factors), these repetitive messages create unnecessary noise in the job log output, making it harder to read and review actual build output.

Proposal

Add a configuration option to suppress or reduce the frequency of "Waiting for pod" messages in the Kubernetes executor. This would help keep job logs cleaner and more focused on actual build output.

Possible Implementation Approaches

1. Complete suppression: Add a boolean flag to disable these messages entirely
2. Reduced frequency: Log the message at longer intervals (e.g., every 30 seconds instead of every poll_interval seconds)
3. Collapsible section: Group these messages into a collapsible section in the job log UI
4. Log level control: Make these messages respect a configurable log verbosity level

Benefits

• Cleaner, more readable job logs
• Reduced log storage for jobs with long pod scheduling delays
• Better signal-to-noise ratio when reviewing job output
• Improved user experience when debugging actual build issues

Configuration Example

[[runners]]
  [runners.kubernetes]
    suppress_pod_wait_messages = true
    # or
    pod_wait_log_interval = "30s"

Related Information

This is particularly relevant for:

• Clusters with autoscaling enabled where pod scheduling can take several minutes
• Resource-constrained environments
• Jobs running during peak usage times

Requested by customer (ZD internal link)