Kernel debug two physical MacBook pro devices

Question

I'm trying to kernel debug a physical Macbook pro device. When my setup contains a VM, between a host MacOS and guest MacOS, its working with no problem using lldb -o "kdp-remote <guest_machine_ip>"

my boot-args on my remote (to debug) Macbook are:

user$ nvram boot-args
boot-args       debug=0x44 kext-dev-mode=1 pmuflags=1 -v kcsuffix=debug

When i'm triggering kernel debug using either sudo dtrace -w -n "BEGIN { breakpoint(); }" or LEFT CMD + RIGHT CMD + PowerButton, the system is in halt mode, not responsive to anything (Mouse not moving, etc) But when I'm trying to connect to the machine from the host using the lldb command provided earlier, it's not working. I've made sure the machines can ping each other and I can set up a SSH connection.

To the best of my understanding, I had to user special Apple adapters (Real™ Ethernet adapter), so I'm using the following adapters to debug the remote machine:

• https://www.apple.com/shop/product/MMEL2AM/A/thunderbolt-3-usb-c-to-thunderbolt-2-adapter
• https://www.apple.com/shop/product/MD463LL/A/thunderbolt-to-gigabit-ethernet-adapter

The physical connections setup looks like that:

MacOS-Debugger --(USBC to Ethernet adapter)--> Ethernet cable <-- (ThunderBolt to Ethernet adapter) <-- (ThunderBolt3 to ThunderBolt2 adapter) <-- MacOS-debuggee

Generally for the debugging I used the following tutorial, which worked for VM debugging: https://knight.sc/debugging/2018/08/15/macos-kernel-debugging.html, I didn't find the exact KDK version, but I don't think it should be the problem here

My debugger is BigSur, and my debugee is Catalina

Any ideas to what the problem may be will be helpful

Answer 1

Your hardware setup sounds fine. Note that you don't necessarily need a direct ethernet cable connection, you can use your normal office-/home-wide wired network, it's fine to have a switch (or even more than one) between debugger and target. The debugger machine can even be using wifi as long as it can reach the target's wired network connection that way, though it's not recommended. The Thunderbolt Ethernet interface need not be Apple branded, but its driver does need to include kernel debugging support. In practice, for example, I've successfully used the Ethernet port in a OWC brand Thunderbolt Dock as well. Using one of Apple's own adapters of course guarantees it will work.

If the boot-args on your target machine are the same as what you used in the VM, that's probably where your problem is coming from. You should specify the device to be used for kernel debugging, using the kdp_match_name= option. Use the ifconfig command in the Terminal to work out which of the listed devices is your Thunderbolt ethernet adapter (enX), possibly based on the IP address. en0 is typically wifi in Mac laptops, so you're usually looking for X > 0.

So you'll end up with something like kdp_match_name=en1 added to your boot-args. Make sure to reboot the machine cleanly before attempting a debugging session after updating the boot-args setting.

A few more comments:

kext-dev-mode=1

This option no longer has any effect. It only existed in OS X 10.10. You can remove it.

I didn't find the exact KDK version, but I don't think it should be the problem here

Having a KDK version that doesn't match the target's exact kernel binary UUID will not prevent a connection, but it will prevent you from doing any meaningful debugging as the memory layout of threads etc. will not be available.

My debugger is Big Sur, and my debuggee is Catalina

If you ever run Big Sur on your target, make sure to add wdt=-1 to your boot-args or the hung/crashed machine will reboot before you get a chance to connect with the debugger.