stack overflow / saved_stack = 0 issue in openbios-qemu

hi!

I've been trying to figure out why FreeBSD-16 crashes when booting in qemu-system-ppc64 w/ machine=mac99 and the supplied openbios.

It /seems/ like the open firmware code in openbios has a saved_stack value of 0x0, putting the stack at the end of the 32 bit address space. Then, when the registers are saved into it before calling of_client_interface it stores r29, r30 and r31 at addresses 0x0, 0x4 and 0x8.

This doesn't crash during freebsd's bootloader or early kernel boot because at that point there's memory mapped at 0x0. However shortly after boot FreeBSD rearranges the VM space so there's nothing mapped at 0x0 (to catch NULL pointers in kernel space) and .. well, things explode pretty quickly after that.

Here's what I'm seeing when I run the bootloader in qemu + gdb

(kgdb) print &saved_stack
$3 = (<data variable, no debug info> *) 0xfff45000
(kgdb) print (void *) saved_stack
$4 = (void *) 0x0
(kgdb)

Right after it loads the bootloader, I check to see what saved_stack is. It's 0x0. I can't find in the code where saved_stack is being explicitly set so I can only assume it's supposed to be high memory.

Then the initialisation happens in of_client_callback:

Dump of assembler code for function of_client_callback:
=> 0x00000000fff025d0 <+0>:     stwu    r1,-16(r1)
   0x00000000fff025d4 <+4>:     stw     r4,8(r1)
   0x00000000fff025d8 <+8>:     mflr    r4
   0x00000000fff025dc <+12>:    stw     r4,4(r1)
   0x00000000fff025e0 <+16>:    lis     r4,-12
   0x00000000fff025e4 <+20>:    addi    r4,r4,20480
   0x00000000fff025e8 <+24>:    lwz     r4,0(r4)

Like the source says, it's saving stuff, then it loads r4 with the contents of "saved_stack", which in this case is 0x0.

   0x00000000fff025ec <+28>:    stwu    r4,-144(r4)
   0x00000000fff025f0 <+32>:    stw     r1,8(r4)
   0x00000000fff025f4 <+36>:    mr      r1,r4

Then it reserves 144 bytes from the stack, stuffs the original stack pointer in there, and then assigns the stack pointer to r4.

At this point, the stack pointer is now 0xffffff70 (144 bytes from the top of memory):

r0             0x2c001f4           46137844
r1             0xffffff70          4294967152
r2             0x0                 0
r3             0x3458abc           54889148
r4             0xffffff70          4294967152

Then it saves all the registers:

   0x00000000fff02678 <+168>:   stw     r28,140(r1)
   0x00000000fff0267c <+172>:   stw     r29,144(r1)
   0x00000000fff02680 <+176>:   stw     r30,148(r1)
   0x00000000fff02684 <+180>:   stw     r31,152(r1)
   0x00000000fff02688 <+184>:   bl      0xfff0e668 <of_client_interface>

And those last three register saves wrap around to effective addresses 0x0, 0x4 and 0x8, overwriting what's there.

Now, I /think/ my analysis is correct - we need to add 12 bytes to SAVED_SPACE and then nothing overwrites / wraps and freebsd boots fine:

diff --git a/arch/ppc/qemu/start.S b/arch/ppc/qemu/start.S
index c679230..4f4678f 100644
--- a/arch/ppc/qemu/start.S
+++ b/arch/ppc/qemu/start.S
@@ -503,7 +503,7 @@ _GLOBAL(saved_stack):
 #define SAVE_SPACE 320
 #else
 #define STKOFF 8
-#define SAVE_SPACE 144
+#define SAVE_SPACE 156
 #endif
 
 GLOBL(of_client_callback):

is that correct? I'll put up a diff/pull request to fix this if so!

Thanks!