Linux on the Mac — state of the union

The MacBook Pro introduction in October caused unusually negative reactions among professional users due to the realization that Apple no longer caters equally to casual and professional customers as it had in the past [YouTube video]. Instead, the company appears to be following an iOS-focused, margin-driven strategy that essentially relegates professionals to a fringe group. This has well-known developers such as Salvatore Sanfilippo (of the Redis project) consider a move back to Linux. Perhaps that's a good moment to look at the current state of Mac hardware support in the kernel. While Macs are x86 systems, they possess various custom chips and undocumented quirks that the community needs to painstakingly reverse-engineer.

GPU switching

Apple is the only remaining vendor to build a multiplexer into hybrid graphics laptops, which have both high-end and low-power GPUs. The multiplexer allows the panel to be switched between the GPUs and the unused GPU to be powered off. All other manufacturers use a "muxless" solution, whereby the discrete GPU is headless and copies rendered data over PCIe into the integrated GPU's framebuffer. "Muxed" solutions, such as the one used by Apple, offer superior power saving and latency, but are more difficult to implement.

The multiplexer built into pre-Retina MacBook Pros is a custom Lattice XP2 FPGA, which is documented to some extent in patent US 8,687,007 B2 [PDF]. Retinas moved to a different display connector (eDP instead of LVDS) to accommodate the higher pixel clock and this forced Apple to come up with a redesign which consists of two chips, a Renesas H8S/2113 controller and a separate off-the-shelf eDP multiplexer from NXP [PDF], TI [PDF], or Pericom [PDF].

Pre-Retina MacBook Pros use a Lattice XP2 (blue) to combine the controller and multiplexer in a single custom chip. Image courtesy of iFixit.

A driver for the controller was written by Seth Forshee in 2012, initially only to control backlight brightness. Andreas Heider subsequently added switching control. Matthew Garrett and Bernhard Frömel contributed code to handle register access on the H8S/2113. (It has the same register layout as its predecessor Lattice XP2, but the registers are accessed via a mailbox rather than directly.)

The panel resolution stored in the video BIOS is notoriously bogus on Macs, so it needs to be probed instead by temporarily switching the DDC lines between GPUs. Attempts to implement this were made by Forshee, Garrett, and Dave Airlie but not pursued into mainline. Development therefore stagnated for about three years. I started another effort in 2015 and the resulting patches that finally enable GPU switching on pre-Retinas were merged into 4.6. A byproduct was documentation on vga_switcheroo and Apple's multiplexer.

The original "unibody" MacBook Pro introduced 2008-2009 with dual NVIDIA GPUs required several extra changes that were figured out by Pierre Moreau. There is one remaining glitch related to hardware acceleration of the console framebuffer that can be worked around by loading nouveau with the nofbaccel option. This machine is no longer supported by Apple since macOS Sierra. Notably, macOS was never capable of switching at runtime on this model or using both GPUs simultaneously, so Linux is squeezing more out of the hardware than Apple ever did.

However a lot still remains to be done: so far, the discrete GPU has to be turned on and off manually by the user. I have begun a set of patches to add runtime power management support which will handle this automatically, however it turned out that a rework of vga_switcheroo audio handling is necessary first, based on Rafael Wysocki's functional dependencies series, which is queued for 4.10. Another unresolved issue is a short flicker during switching if the vertical blanking intervals (VBLANK) of the GPUs happen not to be in sync. Apple's patent talks about the gmux controller lengthening or shortening VBLANK intervals to achieve a seamless switch, but in reality this is not performed by the chip and needs to be done in software instead. Further down on the to-do list is switching while X11 (or Wayland) is running. Currently, switching is only possible on the framebuffer console with no clients connected to the DRM drivers. Airlie has stated that improving this situation is a low priority for him.

Retina MacBook Pros such as this one introduced in October use an H8S/2113 controller (yellow) and accompanying NXP CBTL06142 multiplexer (in-between the GPU and CPU). Image courtesy of iFixit.

While GPU switching on pre-Retinas works, it doesn't yet on Retinas. Once again there is no valid panel resolution stored in the video BIOS, so both GPUs need to probe it. However unlike on pre-Retinas, the AUX channel (which is the DDC equivalent for DisplayPort) is not switchable between GPUs without also switching over the main link. The active GPU therefore has to either cache the panel data or proxy the inactive GPU's access to the panel. Additionally, both GPUs need to link-train their eDP outputs: the DisplayPort specification has a special provision for closed, embedded connections which allows outputs to be set up with a pre-calibrated, known-good drive current and pre-emphasis level. A solution would thus be to have the inactive GPU set up its output with pre-calibrated values determined by the active GPU.

I implemented a few prototypes of this in 2015, which were tested with partial success by Bruno Bierbaumer. Unfortunately Bierbaumer's MacBook Pro suffered an accident (unrelated to the patches) and development has since stagnated. Generally, it is hard to bring up features like this without having the hardware in front of you. Sending patches and dmesg output back and forth only gets one so far. (My own machine is a pre-Retina.) For the time being, Retina users should consider using Bierbaumer's gpu-switch application, which allows selecting the active GPU for the next reboot. The inactive GPU may then be powered down via vga_switcheroo to conserve energy.

Retinas introduced 2013 and onward have an additional requirement for GPU switching: Either the bootloader or the EFI stub in the kernel need to identify as macOS to the firmware, otherwise the integrated GPU is powered down and thus hidden from the operating system. The rationale is that Apple only supports macOS and Windows on its hardware, but never bothered to enable GPU switching on Windows. If the operating system does not identify as macOS, the EFI firmware assumes that it is dealing with Windows and disables various features of the hardware.

This identification scheme is similar in spirit to the ACPI _OSI method but happens before the EFI ExitBootServices() call (i.e. much earlier than the kernel's ACPI subsystem is initialized). It is not necessary on pre-2013 models, which assume Windows when booting in the legacy BIOS mode and macOS otherwise. Users need to be aware that if they expose the integrated GPU on 2013+ models, loading the i915 driver results in an interrupt storm that can be avoided by disabling an ACPI general-purpose event (GPE) via sysfs.

Thunderbolt

Thunderbolt controllers comprise a PCIe switch whose fabric is managed in either of two ways: by a firmware component called Intel Connection Manager (ICM) or natively by the operating system. In the former case, PCI tunnels to newly attached devices are configured in System Management Mode (SMM) behind the operating system's back and the devices appear below ACPI PCI hotplug slots. This is what most vendors do. Apple took the other approach and ships two drivers: an EFI driver that configures devices already attached on boot, and a macOS driver that assumes control after the ExitBootServices() call. When booting Windows, Apple powers the controller off on older machines or reconfigures it at runtime to be controlled by ICM on 2015+ models.

In principle, Apple's approach of foregoing a firmware blob is desirable from a free-software perspective. However Apple's drivers are closed source and Intel hasn't made the Thunderbolt specification public. Andreas Noever took on the Herculean task of reverse-engineering the macOS driver and writing a basic Linux driver. The resulting patches went into 3.17 and initially supported two chips: Cactus Ridge 4C and Falcon Ridge 4C. This year, we have been able to broaden support to further chips:

Light Ridge 4C	(Macs introduced 2011-2012)	supported since 4.7
Eagle Ridge 2C	(MacBook Air introduced 2011)	unsupported, try this patch and report back
Cactus Ridge 4C	(Macs introduced 2012-2013)	supported since 3.17
Falcon Ridge 4C	(Macs introduced 2013-2015)	supported since 3.17
Falcon Ridge 2C	(MacBook Air introduced 2015)	supported since 4.8
Alpine Ridge 4C	(MacBook Pro introduced 2016)	unsupported, try this patch and report back

Thunderbolt controllers consume about 2W even when idle. Apple provides a nonstandard ACPI-based mechanism to power the controller down when nothing is plugged in and I have implemented patches this year to make use of it on Linux. A first preparatory series is in 4.9 to avoid gratuitously waking the controller before and after system sleep. A second preparatory series is queued up for 4.10 to add runtime power management for PCIe Hotplug ports. A third series containing the actual runtime power management for the Thunderbolt controller is slated for 4.11. The patches improve battery life noticeably: idle power consumption on my MacBook Pro drops from 12.2W to 10.5W when powering down Thunderbolt (with the discrete GPU, and AirPort already disabled). macOS achieves 7W as it supports power management on more devices, such as Firewire.

Another upcoming feature is EFI device properties: Thunderbolt controllers possess a 64-bit unique ID, which allows telling them apart when connected together. The ID is stored in a device ROM that the vendor is supposed to burn at the factory. Curiously, Apple skipped that step on Thunderbolt 1 chips and left the device ROM blank with an ID of 0x1000000000000. So how does the controller get a unique ID? It turns out that the EFI driver generates a device ROM with an ID based on the Mac's serial number and communicates it to the macOS driver as a device property.

For Linux 4.10 I submitted patches to retrieve these properties and use the device ROM supplied by EFI in the Thunderbolt driver. The kernel needs to be booted by the EFI stub for this to work. The properties contain a lot more than just the device ROM (e.g. they convey which PCI tunnels were established by the Thunderbolt EFI driver and how the graphics EFI drivers configured the GPUs). See this sample that was generated with the kernel command line option dump_apple_properties. Apple seems to have been using this proprietary protocol for EFI device properties ever since they moved to x86 in 2006, so it took ten years for Linux to catch up. Nevertheless, having that data available now puts us in a much better position to support Macs optimally.

Current work focuses on coping with surprise removal and fixing system sleep-related bugs in the PCIe hotplug driver. Many Thunderbolt features are still needed, such as support for daisy-chaining and establishing DisplayPort-over-Thunderbolt tunnels.

Networking is another area that needs work, specifically it needs a driver for Mac hardware. Thunderbolt 3 is marketed as having a total bandwidth of 40 Gbit/s, which in reality is capped at 32 Gbit/s by the 4x PCIe 3.0 interface of currently available controllers. Still, this promises to rival 25G Ethernet and InfiniBand at a lower price point. The MacBook Pro introduced in October has four Thunderbolt 3 ports which would enable interesting applications like a portable compute cluster with up to five fully-meshed nodes. macOS introduced Ethernet-over-Thunderbolt tunneling in 2013 and Intel ported it to its closed-source Windows driver a year later.

Apparently due to demand from vendors such as Dell, Intel also developed a Linux driver whose source code was surprisingly made public this year and is now at its ninth iteration. However the released code mostly just contains the plumbing between the kernel's networking subsystem and the firmware-based ICM. The real action happens in firmware, which remains closed source. And the dependency on ICM means that the patches only work on non-Macs.

Even so, Intel's driver duplicates a portion of Noever's driver. When asked to eliminate duplicate code and move the remainder into the existing Thunderbolt driver, Intel responded in a somewhat tight-lipped manner that it "does not maintain, develop, and publish Thunderbolt software code running on Apple hardware", which sounds like it came straight out of the legal department to avoid stepping on the toes of their key account in Cupertino. Obviously, having some source code is better than no source code, but Greg Kroah-Hartman has been reluctant to merge the additional driver and has pushed for more review.

Firmware quirks

The EFI firmware on Macs contains a network stack to facilitate downloading macOS recovery images from Apple. A particularly egregious bug is present in the EFI driver for Broadcom 4331 wireless chips built into various 2011 and 2012 Macs as it fails to disable the card upon the ExitBootServices() call. As a result, the card causes an interrupt storm and corrupts memory with DMA transfers of received packets. In principle, this can be used for remote code execution over the air. Garrett discovered the issue in 2012 and sought to fix it with a GRUB quirk, but this only addressed memory corruption and not the interrupt storm, so users continued to see messages such as "irq 17: nobody cared". For Linux 4.7, I submitted an early quirk to reset the wireless card that finally fixes the issue for good. It has also been picked up by all supported stable kernels.

MacBook Pros introduced in 2015 suffer from a similarly annoying issue caused by an unused PCIe root port that Apple forgot to disable in the firmware. The root port is invisible to macOS because it is not enumerated in the ACPI tables, but Linux discovers and initializes it, thereby breaking suspend and power off on these machines. Chen Yu has posted a workaround patch which is not yet in mainline but already included in distributions such as Ubuntu.

SPI input devices and NVMe

Before 2015, mobile Macs used USB to connect the keyboard and trackpad. Newer models equipped with a Force Touch trackpad moved to a custom controller that connects to the southbridge with SPI. On the first model to do so, the MacBook Pro 13" (Early 2015), the controller alternatively supports USB and is switchable between the two interfaces by way of ACPI methods. In USB mode, this model is supported by mainline kernels. All following machines, notably the MacBook 12", leave the USB pins unconnected and are SPI only. Federico Lorenzi has begun reverse-engineering the controller's protocol and developing an experimental out-of-tree driver.

The MacBook 12" also has a custom NVMe controller, which is known to not come out of system sleep, apparently due to missing vendor-specific commands.

In conclusion

Users wishing to try Linux as a dual boot option alongside macOS may want to consider ZFS as it allows cross-mounting between the two operating systems in a stable manner. Other Linux filesystems such as ext4 or Btrfs are unfortunately not well supported on macOS. Vice-versa, HFS Plus on Linux does not support journaling and FileVault2 support is experimental. Kernel developers need however be aware that loading the ZFS modules disables lockdep due to the CDDL taint.

Bringup on Macs is a challenge, but on the bright side we are making huge leaps with every new release. Supporting new hardware generally takes about two years, anything older can be expected to work decently. Battery life is not yet on par with macOS, Thunderbolt lacks many features, and GPU switching only works on pre-Retinas. Despite these limitations, Mac hardware support is significantly ahead on Linux compared to other free operating systems. To quote Apple's classic commercial [YouTube video]: "here's to the crazy ones" who are bringing up Linux on the Mac.