Two comments are rendered after submitting a response

This does not happen always.

Submit a comment in an issue
The placeholder comment is rendered
Once the response is received both the placeholder comment and the response are rendered as a normal response.

This does not happen for system notes, only comments the user made.

I don't know how to reproduce this always, maybe @rspeicher or @blackst0ne can help, both of them reported this on slack as well.