linux-user: madvise() returns 0 on unmapped ranges instead of ENOMEM

Host environment

• Operating system: Arch Linux
• OS/kernel version: 7.1.8-arch1-3
• Architecture: x86_64
• QEMU flavor: qemu-user (I've tested on qemu-x86_64)
• QEMU version: upstream (tested against 0a1ebd44)

Emulated/Virtualized environment

qemu-user

• Architecture: all

Description of problem

Currently, under qemu linux-user, calling madvise() with any hint advice value on a range that is partially or fully unmapped returns 0. This differs from the documented and expected behavior per madvise(2) and per the kernel code, which is to fail with ENOMEM.

Steps to reproduce

Here is a code snippet that is affected by the bug:

#define _GNU_SOURCE
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

int main(void)
{
    int pagesize = getpagesize();

    void *p = mmap(NULL, pagesize, PROT_READ,
                   MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
    assert(p != MAP_FAILED);
    int ret = munmap(p, pagesize);
    assert(ret == 0);

    errno = 0;
    ret = madvise(p, pagesize, MADV_NORMAL);
    int err = errno;
    printf("ret: %d errno: %s (%d)\n", ret, strerrorname_np(err), err);

    return 0;
}

Running locally results in ret: -1 errno: ENOMEM (12) while running under qemu-x86_64 results in ret: 0 errno: 0 (0).

Additional information

The bug is in target_madvise under linux-user/mmap.c which returns 0 for hint advice values without checking if the address range is mapped. I wrote a fix and some test cases and will send the patches soon, referencing this issue.