Multiple user popovers added when repeatedly hovering a user

On current master I'm seeing multiple user popups. If you mouseover/out a few times there are extra shadows:

And this in devtools:

I think related to !23904 (merged) - reverting 11680481 and it doesn't happen