Draft: fix/workaround CI's UI test
This doesn't completely fix #2658, but it works around it. Ideally this would be done in pytest, but I couldnt' figure out how to get the imports to work correctly since this check depends on being able to detect changed packages (and the python stuff for that is in .ci/lib.