This is related to #531
Per @kraxel: EHCI could be relatively easy to fix that way because most of the actual work already happens in the frame timer callback.