Abort in audio_calloc() of ac97

Host environment

• Operating system: Ubuntu 20.04
• OS/kernel version: 5.15.0-56-generic
• Architecture: x86
• QEMU flavor: qemu-system-x86_64
• QEMU version: 7.0.94
• QEMU command line: see below

Emulated/Virtualized environment

• Operating system: Buildroot Linux
• OS/kernel version: 5.15.18
• Architecture: x86

Description of problem

Section 5.10.2 of the AC97 specification (https://hands.com/~lkcl/ac97_r23.pdf) shows the feasibility to support for rates other than 48kHZ. Specifically, AC97_PCM_Front_DAC_Rate (reg 2Ch) should be from 8kHZ to 48kHZ.

An adversary can leverage this to crash QEMU.

A nornal 48kHZ setting is like this.

ac97_realize
  open_voice
    as->freq = 0xbb80 # 0xbb80=48000
    AUD_open_out
      audio_pcm_create_voice_pair_out (sw is NULL)
        audio_pcm_sw_init_out
          sw->info.freq = as->freq (in audio_pcm_init_info())
          sw->ratio = ((int64_t) sw->hw->info.freq << 32) / sw->info.freq
          samples = ((int64_t) sw->HWBUF->size << 32) / sw->ratio (in audio_pcm_sw_alloc_resources_out())

A non-48kHZ setting is like this. Since as->freq is too small, sw->ratio is too large. Finally, samples is zero, failing the audio_calloc() in audio_pcm_sw_alloc_resources_out().

nam_writew
  open_voice
    as->freq = 0x6
    AUD_open_out
      audio_pcm_sw_init_out (sw is not NULL)
        sw->info.freq = as->freq (in audio_pcm_init_info())
        sw->ratio = ((int64_t) sw->hw->info.freq << 32) / sw->info.freq
        samples = ((int64_t) sw->HWBUF->size << 32) / sw->ratio (in audio_pcm_sw_alloc_resources_out())
        audio_calloc(.., samples, ) (in audio_pcm_sw_alloc_resources_out())

Steps to reproduce

1. download the prepared rootfs and the image.

   https://drive.google.com/file/d/1IfVCvn76HY-Eb4AZU7yvuyPzM3QC1q10/view?usp=sharing https://drive.google.com/file/d/1JN6JgvOSI5aSLIdTEFKiskKbrGWFo0BO/view?usp=sharing

2. run the following script.

QEMU_PATH=../../../qemu-devel/build/x86_64-softmmu/qemu-system-x86_64
KERNEL_PATH=./bzImage
ROOTFS_PATH=./rootfs.ext2
$QEMU_PATH \
    -M q35 -m 1G \
    -kernel $KERNEL_PATH \
    -drive file=$ROOTFS_PATH,if=virtio,format=raw \
    -append "root=/dev/vda console=ttyS0" \
    -net nic,model=virtio -net user \
    -device ac97,audiodev=snd0 -audiodev none,id=snd0 \
    -nographic

3. with spawned shell (the user is root and the password is empty), run ac97-00.

Additional information

In the latest QEMU, this issue was generally fixed by 12f4abf6 and 0cbc8bd4 that remove abort() from the source code. Even though, I still plan to send a patch so that the warning about the invalid freq will be gone.