now | writings | rss | github | twitter | contact

OpenBSD in Stereo with Linux VFIO

posted to writings on nov 12th, 2018 with tags linux, nerd, and openbsd and commented on seven times

I use a Huawei Matebook X as my primary OpenBSD laptop and one aspect of its hardware support has always been lacking: audio never played out of the right-side speaker. The speaker did actually work, but only in Windows and only after the Realtek Dolby Atmos audio driver from Huawei was installed. Under OpenBSD and Linux, and even Windows with the default Intel sound driver, audio only ever played out of the left speaker.

Now, after some extensive reverse engineering and debugging with the help of VFIO on Linux, I finally have audio playing out of both speakers on OpenBSD.

VFIO

The Linux kernel has functionality called VFIO which enables direct access to a physical device (like a PCI card) from userspace, usually passing it to an emulator like QEMU.

To my surprise, these days, it seems to be primarily by gamers who boot Linux, then use QEMU to run a game in Windows and use VFIO to pass the computer's GPU device through to Windows.

By using Linux and VFIO, I was able to boot Windows 10 inside of QEMU and pass my laptop's PCI audio device through to Windows, allowing the Realtek audio drivers to natively control the audio device. Combined with QEMU's tracing functionality, I was able to get a log of all PCI I/O between Windows and the PCI audio device.

Using VFIO

To use VFIO to pass-through a PCI device, it first needs to be stubbed out so the Linux kernel's default drivers don't attach to it. GRUB can be configured to instruct the kernel to ignore the PCI audio device (8086:9d71) and explicitly enable the Intel IOMMU driver by adding the following to /etc/default/grub and running update-grub:

GRUB_CMDLINE_LINUX_DEFAULT="text pci-stub.ids=8086:9d71 iommu=pt intel_iommu=on"

With the audio device stubbed out, a new VFIO device can be created from it:

sudo modprobe pci-stub
sudo modprobe vfio-pci

echo 0000:00:1f.3 | sudo tee /sys/bus/pci/devices/0000:00:1f.3/driver/unbind
echo 0x8086 0x9d71 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

Then the VFIO device (00:1f.3) can be passed to QEMU:

sudo qemu-img create -f qcow2 -b win10.img win10-tmp.img

sudo ../qemu/x86_64-softmmu/qemu-system-x86_64 \
    -M q35 -m 2G -cpu host,kvm=off \
    -enable-kvm \
    -device vfio-pci,host=00:1f.3,multifunction=on,x-no-mmap \
    -hda win10-tmp.img \
    -trace events=events.txt 2>&1 | tee debug-output

I was using my own build of QEMU for this, due to some custom logging I needed (more on that later), but the default QEMU package should work fine. The events.txt was a file of all VFIO events I wanted logged (which was all of them).

Since I was frequently killing QEMU and restarting it, Windows 10 wanted to go through its unexpected shutdown routine each time (and would sometimes just fail to boot again). To avoid this and to get a consistent set of logs each time, I used qemu-img to take a snapshot of a base image first, then boot QEMU with that snapshot. The snapshot just gets thrown away the next time qemu-img is run and Windows always starts from a consistent state.

QEMU will now log each VFIO event which gets saved to a debug-output file.

[...]
9645@1541992466.382461:vfio_pci_read_config  (0000:00:1f.3, @0x2e, len=0x2) 0x3200
9645@1541992466.395726:vfio_region_read  (0000:00:1f.3:region0+0xc, 2) = 0x0
9645@1541992466.395792:vfio_region_read  (0000:00:1f.3:region0+0xe, 2) = 0x1
9645@1541992466.396021:vfio_region_write  (0000:00:1f.3:region0+0xc, 0x0, 2)
[...]

With a full log of all PCI I/O activity from Windows, I compared it to the output from OpenBSD and tried to find the magic register writes that enabled the second speaker. After days of combing through the logs and annotating them by looking up hex values in the documentation, diffing runtime register values, and even brute-forcing it by mechanically duplicating all PCI I/O activity in the OpenBSD driver, nothing would activate the right speaker.

One strange thing that I noticed was if I booted Windows 10 in QEMU and it activated the speaker, then booted OpenBSD in QEMU without resetting the PCI device's power in-between (as a normal system reboot would do), both speakers worked in OpenBSD and the configuration that the HDA controller presented was different, even without any changes in OpenBSD.

A Primer on Intel HDA

Most modern computers with integrated sound chips use an Intel High Definition Audio (HDA) Controller device, with one or more codecs (like the Realtek ALC269) hanging off of it. These codecs do the actual audio processing and communicate with DACs and ADCs to send digital audio to the connected speakers, or read analog audio from a microphone and convert it to a digital input stream. In my Huawei Matebook X, this is done through a Realtek ALC298 codec.

On OpenBSD, these HDA controllers are supported by the azalia(4) driver, with all of the per-codec details in the lengthy azalia_codec.c file. This file has grown quite large with lots of codec- and machine-specific quirks to route things properly, toggle various GPIO pins, and unmute speakers that are for some reason muted by default.

azalia0 at pci0 dev 31 function 3 "Intel 200 Series HD Audio" rev 0x21: msi
azalia0: host: High Definition Audio rev. 1.0
azalia0: host: 9 output, 7 input, and 0 bidi streams
azalia0: found a codec at #0
azalia0: found a codec at #2
azalia_init_corb: CORB allocation succeeded.
azalia_init_corb: CORBWP=0; size=256
azalia_init_rirb: RIRB allocation succeeded.
azalia_init_rirb: RIRBRP=0, size=256
azalia0: codec[0] vid 0x10ec0298, subid 0x320019e5, rev. 1.3, HDA version 1.0
azalia_codec_init: There are 36 widgets in the audio function.
[...]
azalia0: codecs: Realtek ALC298, Intel/0x280b, using Realtek ALC298

The azalia driver talks to the HDA controller and sets up various buffers and then walks the list of codecs. Each codec supports a number of widget nodes which can be interconnected in various ways. Some of these nodes can be reconfigured on the fly to do things like turning a microphone port into a headphone port.

The newer Huawei Matebook X Pro released a few months ago is also plagued with this speaker problem, although it has four speakers and only two work by default. A fix is being proposed for the Linux kernel which just reconfigures those widget pins in the Intel HDA driver. Unfortunately no pin reconfiguration is enough to fix my Matebook X with its two speakers.

While reading more documentation on the HDA, I realized there was a lot more activity going on than I was able to see through the PCI tracing.

For speed and efficiency, HDA controllers use a DMA engine to transfer audio streams as well as the commands from the OS driver to the codecs. In the output above, the CORBWP=0; size=256 and RIRBRP=0, size=256 indicate the setup of the CORB (Command Output Ring Buffer) and RIRB (Response Input Ring Buffer) each with 256 entries. The HDA driver allocates a DMA address and then writes it to the two CORBLBASE and CORBUBASE registers, and again for the RIRB.

When the driver wants to send a command to a codec, such as CORB_GET_PARAMETER with a parameter of COP_VOLUME_KNOB_CAPABILITIES, it encodes the codec address, the node index, the command verb, and the parameter, and then writes that value to the CORB ring at the address it set up with the controller at initialization time (CORBLBASE/CORBUBASE) plus the offset of the ring index. Once the command is on the ring, it does a PCI write to the CORBWP register, advancing it by one. This lets the controller know a new command is queued, which it then acts on and writes the response value on the RIRB ring at the same position as the command (but at the RIRB's DMA address). It then generates an interrupt, telling the driver to read the new RIRBWP value and process the new results.

Since the actual command contents and responses are handled through DMA writes and reads, these important values weren't showing up in the VFIO PCI trace output that I had gathered. Time to hack QEMU.

Logging DMA Memory Values in QEMU

Since DMA activity wouldn't show up through QEMU's VFIO tracing and I obviously couldn't get Windows to dump these values like I could in OpenBSD, I could make QEMU recognize the PCI write to the CORBWP register as an indication that a command has just been written to the CORB ring.

My custom hack in QEMU adds some HDA awareness to remember the CORB and RIRB DMA addresses as they get programmed in the controller. Then any time a PCI write to the CORBWP register is done, QEMU fetches the new CORB command from DMA memory, decodes it into the codec address, node address, command, and parameter, and prints it out. When a PCI read of the RIRBWP register is requested, QEMU reads the response and prints the corresponding CORB command that it stored earlier.

With this hack in place, I now had a full log of all CORB commands and RIRB responses sent to and read from the codec:

9645@1541992466.588081:vfio_region_read  (0000:00:1f.3:region0+0x48, 2) = 0xdb
CORBWP advance to 220, last WP 219
CORB[220] = 0x21f0800 (caddr:0x0 nid:0x21 control:0xf08 param:0x0)
9645@1541992466.588109:vfio_region_write  (0000:00:1f.3:region0+0x48, 0xdc, 2)
[...]
9645@1541992466.588386:vfio_region_write  (0000:00:1f.3:region0+0x5d, 0x1, 1)
RIRBWP advance to 220, last WP 219
CORB caddr:0x0 nid:0x21 control:0xf08 param:0x0 response:0x82 (ex 0x0)
9645@1541992466.588431:vfio_region_read  (0000:00:1f.3:region0+0x58, 2) = 0xdc
[...]

An early version of this patch left me stumped for a few days because, even after submitting all of the same CORB commands in OpenBSD, the second speaker still didn't work. It wasn't until re-reading the HDA spec that I realized the Windows driver was submitting more than one command at a time, writing multiple CORB entries and writing a CORBWP value that was advanced by two. This required turning my CORB/RIRB reading into a for loop, reading each new command and response between the new CORBWP/RIRBWP value and the one previously seen.

Sure enough, the magic commands to enable the second speaker were sent in these periods where it submitted more than one command at a time.

Minimizing the Magic

The full log of VFIO PCI activity from the Windows driver was over 65,000 lines and contained 3,150 CORB commands, which is a lot to sort through. It took me a couple more days to reduce that down to a small subset that was actually required to activate the second speaker, and that could only be done through trial and error:

  • Boot OpenBSD with the full list of CORB commands in the azalia driver
  • Comment out a group of them
  • Compile kernel and install it, halt the QEMU guest
  • Suspend and wake the laptop, resetting PCI power to the audio device to reset the speaker/Dolby initialization and ensure the previous run isn't influencing the current test (I'm guessing there is an easier to way to reset PCI power than suspending the laptop, but oh well)
  • Start QEMU, boot OpenBSD with the new kernel
  • Play an MP3 with mpg123 which has alternating left- and right-channel audio and listen for both channels to play

This required a dozen or so iterations because sometimes I'd comment out too many commands and the right speaker would stop working. Other times the combination of commands would hang the controller and it wouldn't process any further commands. At one point the combination of commands actually flipped the channels around so the right channel audio was playing through the left speaker.

The Result

After about a week of this routine, I ended up with a list of 662 CORB commands that are needed to get the second speaker working. Based on the number of repeated-but-slightly-different values written with the 0x500 and 0x400 commands, I'm guessing this is some kind of training data and that this is doing the full Dolby/Atmos system initialization, not just turning on the second speaker, but I could be completely wrong.

In any case, the stereo sound from OpenBSD is wonderful now and I can finally stop downmixing everything to mono to play from the left speaker. In case you ever need to do this, sndiod can be run with -c 0:0 to reduce the channels to one.

Due to the massive size of the code needed for this quirk, I'm not sure if I'll be committing it upstream in OpenBSD or just saving it for my own tree. But at least now the hardware support chart for my Matebook is all yeses for the things I care about.

I've also updated the Linux bug report that I opened before venturing down this path, hoping one of the maintainers of that HDA code that works at Intel or Realtek knew of a solution I could just port to OpenBSD. I'm curious to see what they'll do with it.


Thanks to rjc for proofreading and feedback.

Comments? Contact me via Twitter or e-mail.

7 Comments

Emmanuel Vadot (authentic, via ) on november 13th, 2018 at 13:40:25:

Impressive, very interesting read, thanks

Daniel (authentic, via ) on november 13th, 2018 at 20:35:28:

Wow. Iā€™m tweaking FreeBSD to also get an all YES list for my Matebook X Pro. Still a long way for me to go but already learned a lot from your Matebook X posts. Thanks a lot for sharing.

Yuki (authentic, via ) on november 13th, 2018 at 22:04:51:

What.

Yuki (authentic, via ) on november 13th, 2018 at 22:07:20:

Okay, my brain is kinda awake now.
So, you're using Linux as the host and try to get the audio to work nicely by running a Windows VM VFIO

Pietro (authentic, via ) on november 14th, 2018 at 17:55:46:

Serious hacking shit. Well done

Leo Unglaub (authentic, via ) on november 16th, 2018 at 14:27:13:

Awesome work. This workflow could be used for other stuff as well. Some WiFi cards come to mind. Very nice work, hopefully it will find its way into the main kernel.

šŸ™‡šŸ»ā€ā™‚ļø (authentic, via ) on november 19th, 2018 at 12:17:07:

wow!!! cool :-)
nice work!