GPG verification broken when changing account email

I changed the primary email in my account (just changed the role of primary and verified email), and now all my previously signed commits appear with:

Verified commit with unverified email

This commit was previously signed with a verified signature and verified committer email address. However the committer email address is no longer verified to the same user.

If the signature is valid for an email that is associated with the account and verified, it should still appear as valid.