Unmark lists from "Add List" does not remove list from board

Summary

I have lists on a board for every label in the project. These got added for some reason but cannot be removed.

I expect to be able to remove them by:

• Clicking the "Add List" button
• Unchecking the list/label that is about

Nothing happens. There seems to not be another way to remove lists from a board.