The history on this one is complicated.  The note to README was added in
commit 1f95747a ("Add openssl-dev and xorg-x11-dev to README") in 2007.

At the time, there was a vendered version of Qemu in xen.git with a local
modification using <X11/keysymdef.h> to access the monitor console over VNC.

The final reference to keysymdef.h was dropped in commit 85896a7c ("build:
add autoconf to replace custom checks in tools/check") in 2012.  The next
prior mention was in 2009 with commit a8ccb671

 ("tools: fix x11 check")
noting that x11 was not a direct dependcy of Xen; it was transitive through
SDL for Qemu for source-based distros.

It appears there may have been other unspecified dependencies on xorg,
e.g. the use of lndir by unmodified_drivers which are no longer relevant
either.

These days its only the Debian based dockerfiles which install xorg-x11, and
Qemu builds fine in these and others without x11.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Release-Acked-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>