Prioritization of Idle Count Over Pending Jobs in Fleeting + Taskscaler

Status update 2024-12-02

• The taskscaler/fleeting library supports idle_count with preemptive mode disabled, where jobs will be requested regardless of whether there's any idle capacity. This makes having idle capacity just quicker when there isn't.

• A documentation update is required to make this clear.

Summary

Currently, the Runner Fleet module prioritizes the idle_count over pending CI/CD jobs when operating in preemptive mode with fleeting and taskscaler. If idle_count > 0, the runner manager does not request new jobs until idle instances are utilized. While this approach ensures immediate job execution when idle instances are available, it becomes counterintuitive in autoscaling environments, where pending jobs remain unassigned, leading to delays in pipeline execution.

Problem Statement:

• In autoscaling scenarios, prioritizing idle instances over pending jobs increases pipeline duration.
• When a large number of jobs are queued, the pending jobs are delayed unnecessarily, as the runner manager waits for idle instances to be assigned first.
• This behavior results in a queue bottleneck, reducing overall system efficiency.

Proposed Solution:

• Adjust the logic to balance prioritization between idle instances and pending jobs.
• Introduce a threshold mechanism where pending jobs are assigned without waiting if the queue exceeds a certain limit, even if idle instances are available.
• Provide a configurable toggle for users to define their preferred prioritization strategy.