Fix #1971 "Backtrace on stormkeep"

Happens on stormkeep because walls are spawned and removed because they're for race, but the intrusive list isn't seeing the removal. The next entity to spawn gets the same entity number, but the list already has that number as being used.