Spike: Consider Separating running and scheduled jobs into a separate tables
We reached a limit on optimizing the StuckCiJobsWorker running and scheduled_at queries which timeout. In one MR we discussed adding tables per status for running similar to what we did for pending builds:
Although I'm not sure if there's a straightforward way to decrease the timing, adding another index might not work as expected due to data distribution (and it's also something we're trying to avoid). We probably need a long-term solution like intermediate tables.
This spike should determine whether this makes sense or if Time Decay alone can resolve some of the pain around the StuckCiJobsWorker (perhaps we could partition by status and then date within status):