Polling loop reference implementation
As part of the polling loop SIG meeting, we would like to implement an improved polling loop reference implementation.
The main complaints about the existing polling loop, is that it prioritises keeping the prescribed polling rate too much. If the polling thread cannot keep up, it skips work items. What's more, it will consistently skip the same work items effectively dropping them from the polling loop entirely.
This strategy doesn't work well on non-real time systems, especially in environments under heavy load where the polling thread can go longs periods of time without being serviced. For example, CI runners and a kubernetes cluster.
A new reference implementation should treat the list of work items to poll as more important than the polling period. That is if it cannot keep up, it should still service every work item and just complain with a log that it is not able to keep up. We should never send the "API_PollThreadOutOfSync" exceptions to clients, we should just try our best to do what was configured.