Fix transient remove issue from board and see milestone removed spec

What does this MR do?

Fix transient remove issue from board and see milestone removed spec.

Are there points in the code the reviewer needs to double check?

No longer relying on UI state to be up to date as this is testing the board, not issue page.

Why was this MR needed?

Transient spec failure, https://gitlab.com/gitlab-org/gitlab-ee/builds/18775957

Does this MR meet the acceptance criteria?

• Tests
  • Added for this feature/bug
  • All builds are passing
• Conform by the merge request performance guides
• Conform by the style guides
• Branch has no merge conflicts with master (if it does - rebase it please)
• Squashed related commits together

What are the relevant issue numbers?

Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/2660