Open
Milestone

Fix XEN_SYSCTL_numainfo[node].memsize for IO memory holes

Summary

When XEN_SYSCTL_numainfo reports the memsize of NUMA nodes, it includes pages mapping MMIO regions.

Issue details

Specifically, XEN_SYSCTL_numainfo[NUMA node n].memsize returns NODE_DATA(nid)->node_spanned_pages, but this span includes MMIO regions that are not defined as e820 RAM.

This is a bug in the implementation of XEN_SYSCTL_numainfo and was noted when adding this Hypercall command:

https://listman.redhat.com/archives/libvir-list/2013-July/081186.html

→ This needs to be fixed to get correct NUMA node memory in xl info -n, hwloc, and when used, by the Xapi toolstack.

Proposed fix:

Fix the NUMA node memsize reported by XEN_SYSCTL_numainfo:

  • Add a new entry to the NODE_DATA() struct: node_present_pages (Linux uses the same variable)
    • Add a minimal implementation for x86 based on the E820_RAM entries in the NUMA node's memory span.
    • Initially, this x86 implementation assumes (like the memory scrubbing code as well) the following:
      • NUMA node memory spans are not interleaved: Possible memory holes shall not include memory of other NUMA nodes.
    • A more advanced implementation may use memory blocks (would need more investigation, to be discussed).
  • During the Xen hypervisor NUMA setup, use E820_RAM ranges to initialize NODE_DATA(nid)->node_present_pages.
  • In XEN_SYSCTL_numainfo, use it to return the NUMA node's memsize.
  • Update NODE_DATA(nid)->node_present_pages on memory hot-add.

To-dos:

  • Early Reviews
  • Test on many different machines
  • Work with the ARM NUMA maintainer on approval and the corresponding ARM implementation in the setup code and memory hot-add/remove
  • Review on xen-devel

Patches

  • With the NUMA work for ARM, generic NUMA code was extracted from x86 after 4.17.
  • Therefore, the patches for 4.17 and master (for xen-devel) need to patch different files and need to have an extra define in the generic numa.h

Review

  • Two separate reviews (one fore each branch) are linked in the Merge Requests tab below.
  • Of course, for the formal review on xen-devel, a patch will be sent.
  • To check the code locally, you can download the patches (or diffs) using GitLab's "Code" button (top-right corner) in the Merge Requests