Skip to content

Resolve "Troubleshoot unreliable airflow behaviour"

Thomas La Piana requested to merge 55-troubleshoot-unreliable-airflow-behaviour into master

Closes #55 (closed)

If i had to guess it is related to the scheduler not being able to handle keeping track of that many tasks... Check webserver and scheduler settings to make sure they're optimal

Changes

Tweaked settings to decrease CPU usage by the webserver and scheduler. Had 6 vCPUs provisioned but had parallelism set at 8, along with the scheduler having 2 threads and the webserver i suspect this is what was causing the weird behaviour. Lowered parallelism back down to 6 and cut the CPU usage of the webserver and scheduler. Will adjust further if needed but this should be stable

Edited by Thomas La Piana

Merge request reports