file: Add an internal "mode"

Previously we relied on the implicit assumption filename xor directory,
representing two modes.  Make this explicit with an internal mode
variable.

This is just refactoring and should not change the functionality.
However we're now more strict about duplicate file= or dir= parameters
appearing on the command line.  Previously only the last one had an
effect and the others were silently ignored.  Now we give an error in
this case.  eg this worked before but now gives an error:

  $ ./nbdkit file /var/tmp /var/tmp/fedora-36.img
  nbdkit: error: file|dir parameter can only appear once on the command line