Withdrawn Invitations Remain For The Recepient

When creating a direct message/invitation, and then deleting it, the recipient cannot reject or accept, and it is stuck.

The logs show this:

WARN conduit::api::client_server::membership: Failed to leave room @user:matrix.domain.com remotely: M_BAD_STATE: User is not invited.