Skip to content

Run _do_cleanup_batch in separate thread to unblock metric publishing

Jeremiah Bonney requested to merge jbonney/cleanup-metric-fix into master

Description

This MR fixes the issue with metrics appearing to not be/sporadically being published when running cleanup. The reason for their delay is because both the metric publishing and _cleanupWorker are on the same event loop and don't play nice. The expensive _do_cleanup_batch method is a synchronous method, which blocks until it's completed. Also, I don't see any await usage within _cleanupWorker outside of the await asyncio.sleep after cleanup is finished, so I'm pretty sure no metrics can get published until that sleep.

To get around this, use run_in_executor to run the expensive _do_cleanup_batch in a separate thread, and await on the result. This frees up the metrics to get published ASAP.

I tested this using docker-compose-cleanup.yml and enabled metrics, and before this change saw that metrics only got published between cleanup iterations. With this change, the metrics are no longer blocked and are processed/published right away.

Merge request reports

Loading