## Host environment - Operating system: Ubuntu, Linux 7.0, x86-64 (AMD Ryzen 9 9950X) - QEMU: master at a1c27ca02bcfeea7deb6d4634c70d5b3b421a76d (11.1.50), configured with `--target-list=x86_64-linux-user,x86_64-softmmu`; also 10.2.1 (Debian 1:10.2.1+ds-1ubuntu3.2) - QEMU flags: none for linux-user (`qemu-x86_64 ./farstack`); for system mode, `-accel tcg -machine q35 -m 256M -nographic -no-reboot -kernel bzImage -initrd initramfs.cpio` ## Emulated CPU Default (`qemu64`), TCG. ## Description of problem In 64-bit mode the stack pointer is RSP, 64 bits wide, and SS's base and B bit play no part, whatever the operand size of the instruction (Intel SDM Vol. 1, the stack's address size in 64-bit mode; AMD APM Vol. 2, segment bases in 64-bit mode). A far RET (`lretl`), a far CALL through an m16:32 pointer, or an IRET (`iretl`) with a 32-bit operand size in 64-bit code pops or pushes 4-byte slots at RSP and moves RSP by their size. QEMU uses RSP masked by SS's B bit (to 32 or 16 bits), plus SS's base, for these when the operand size is 16 or 32 bits. On an ordinary Linux stack, above 4 GiB, the access goes to the low 32 bits of RSP and the process dies of SIGSEGV. With SS's base nonzero (an LDT data segment), the access goes to that base plus the masked RSP. The final `SET_ESP` then sets RSP to the 32-bit value zero-extended, or for a 16-bit B bit only SP. `helper_ret_protected` (far RET and IRET) in `target/i386/tcg/seg_helper.c` treats only the 64-bit operand size as 64-bit mode: ```c #ifdef TARGET_X86_64 if (shift == 2) { sa.sp_mask = -1; } else #endif { sa.sp_mask = get_sp_mask(env->segs[R_SS].flags); } sa.sp = env->regs[R_ESP]; sa.ss_base = env->segs[R_SS].base; ``` `helper_lcall_protected` does the same for a far CALL to a code segment. The 16- and 32-bit cases there take `get_sp_mask` and SS's base, and the code says so: `/* XXX: check 16/32 bit cases in long mode */`. Even the 64-bit RET path adds SS's base. The helpers are shared, so `qemu-system-x86_64` under TCG is affected too. A user process in a guest gets SIGSEGV where it does not under KVM. ## Steps to reproduce 1. Save the program below as `farstack.c` and build it on an x86-64 host: `gcc -O2 -static -no-pie -o farstack farstack.c` (`-no-pie` so the return targets fit the 32-bit offsets.) 2. Run `./farstack` natively: all three lines say `correct`, exit status 0. 3. Run `qemu-x86_64 ./farstack`: ``` stack at 0x7cd02bbfdde4 qemu: uncaught target signal 11 (Segmentation fault) - core dumped lretl : killed by signal 11 (Segmentation fault): wrong qemu: uncaught target signal 11 (Segmentation fault) - core dumped lcall m16:32 : killed by signal 11 (Segmentation fault): wrong qemu: uncaught target signal 11 (Segmentation fault) - core dumped iretl (to ESP) : killed by signal 11 (Segmentation fault): wrong ``` As the init of a minimal Linux 7.2.6 guest, `qemu-system-x86_64 -accel tcg` gives the same three SIGSEGVs, and `-accel kvm` prints `correct` three times. ```c /* In 64-bit mode the stack pointer is RSP, 64 bits, whatever the operand * size of a far CALL, far RET or IRET, and SS's base and B bit play no * part. Each case below runs one with a 32-bit operand size on the * process's own stack (above 4 GiB) and reports how far RSP moved. * Build: gcc -O2 -static -no-pie -o farstack farstack.c * Run: ./farstack, natively and under qemu-x86_64. */ #include <signal.h> #include <stdio.h> #include <string.h> #include <sys/wait.h> #include <unistd.h> long retf32(void), lcall32(void), iret32(void); char iret_stack[64] __attribute__((aligned(16))); /* lcall32's m16:32 far pointer; main fills in CS. */ struct __attribute__((packed)) { unsigned int off; unsigned short sel; } far_ptr; __asm__(".text\n" /* lretl: pops a 4-byte EIP and a 4-byte CS slot. RSP moves by 0. */ "retf32: mov %rsp, %rdx\n" " sub $8, %rsp\n" " movl $1f, (%rsp)\n" " mov %cs, %eax\n" " movl %eax, 4(%rsp)\n" " lretl\n" "1: mov %rsp, %rax\n" " sub %rdx, %rax\n" " mov %rdx, %rsp\n" " ret\n" /* lcall through m16:32: pushes CS and EIP as 4 bytes each. RSP moves by -8 at the target. */ "lcall32: mov %rsp, %rdx\n" " lcall *far_ptr(%rip)\n" "far_target: mov %rsp, %rax\n" " sub %rdx, %rax\n" " mov %rdx, %rsp\n" " ret\n" /* iretl: pops EIP, CS, EFLAGS, ESP and SS as 4 bytes each; RSP becomes the popped ESP, iret_stack's address. */ "iret32: mov %rsp, %rdx\n" " pushfq\n" " pop %rcx\n" " sub $20, %rsp\n" " movl $1f, (%rsp)\n" " mov %cs, %eax\n" " movl %eax, 4(%rsp)\n" " movl %ecx, 8(%rsp)\n" " movl $iret_stack + 64, 12(%rsp)\n" " mov %ss, %eax\n" " movl %eax, 16(%rsp)\n" " iretl\n" "1: mov %rsp, %rax\n" " mov %rdx, %rsp\n" " ret\n"); extern char far_target[]; static int run(const char *name, long (*f)(void), long want) { fflush(stdout); pid_t p = fork(); if (p == 0) { long got = f(); printf("%s: RSP moved by %ld, expected %ld: %s\n", name, got, want, got == want ? "correct" : "wrong"); fflush(stdout); _exit(got != want); } int st; waitpid(p, &st, 0); if (WIFSIGNALED(st)) { printf("%s: killed by signal %d (%s): wrong\n", name, WTERMSIG(st), strsignal(WTERMSIG(st))); return 1; } return WEXITSTATUS(st); } static long iret32_moved(void) { return iret32() - (long)(iret_stack + 64); } int main(void) { int x, bad = 0; unsigned short cs; __asm__("mov %%cs, %0" : "=r"(cs)); far_ptr.off = (unsigned int)(unsigned long)far_target; far_ptr.sel = cs; printf("stack at %p\n", (void *)&x); bad |= run("lretl ", retf32, 0); bad |= run("lcall m16:32 ", lcall32, -8); bad |= run("iretl (to ESP) ", iret32_moved, 0); return bad; } ``` ## Additional information A possible fix: decide the stack by the mode (`HF_CS64_MASK`), not by the operand size, in both helpers. The operand size still decides the size of the slots. ```diff --- a/target/i386/tcg/seg_helper.c +++ b/target/i386/tcg/seg_helper.c @@ void helper_lcall_protected(CPUX86State *env, int new_cs, target_ulong new_eip, #ifdef TARGET_X86_64 - /* XXX: check 16/32 bit cases in long mode */ if (shift == 2) { @@ { sa.sp = env->regs[R_ESP]; - sa.sp_mask = get_sp_mask(env->segs[R_SS].flags); - sa.ss_base = env->segs[R_SS].base; +#ifdef TARGET_X86_64 + if (env->hflags & HF_CS64_MASK) { + /* In 64-bit mode the stack is RSP, whatever the size. */ + sa.sp_mask = -1; + sa.ss_base = 0; + } else +#endif + { + sa.sp_mask = get_sp_mask(env->segs[R_SS].flags); + sa.ss_base = env->segs[R_SS].base; + } @@ static inline void helper_ret_protected(CPUX86State *env, int shift, #ifdef TARGET_X86_64 - if (shift == 2) { + if (env->hflags & HF_CS64_MASK) { + /* In 64-bit mode the stack is RSP, whatever the operand size. */ sa.sp_mask = -1; + sa.ss_base = 0; } else #endif { sa.sp_mask = get_sp_mask(env->segs[R_SS].flags); + sa.ss_base = env->segs[R_SS].base; } sa.sp = env->regs[R_ESP]; - sa.ss_base = env->segs[R_SS].base; ``` (The full patch is attached; checkpatch.pl reports no problems.) A 64-bit operand size only exists in 64-bit mode, so the 64-bit cases keep a 64-bit stack. The far CALL path through a call gate is not changed. With this change on master, the reproducer prints `correct` three times under both `qemu-x86_64` and `qemu-system-x86_64 -accel tcg`. A randomized comparison of far CALL, far RET and IRET in every operand size, after segment loads that leave SS with a base or a 16-bit B bit, agrees with the semantics model under `qemu-x86_64` (5000 tests per form) and `qemu-i386`. That comparison also covers about a million tests across all integer forms in each mode. It used the fixes for two related bugs reported separately: `modify_ldt`'s struct size, and a 16-bit SS aborting `qemu-x86_64`. Found while validating an x86 semantics model against QEMU and real AMD hardware.