Remove old rules when they don't exist anymore
By the way our recipe work it only appends files to the folder, we need to clean the rules folder before writing the new rules to cleanup.
Else we have no way to remove old alerts or to just rename them.