I need to interpose (get my functions called instead of the original functions) some OS X system calls to overcome a flaw in a piece of closed-source software.
Preferably, the resulting solution would work under 10.5 (Leopard) and newer, but I might be able to require 10.6 (Snow Leopard) if the argument were strong enough.
Preferably, the resulting solution would be an executable, but I might settle for a script.
Preferably, the resulting solution would be able to interpose ("steal the vectors") even after the target application is running, but I could settle for a technology that must inject itself as the application is loading.
Preferably, the resulting solution would be developed in C or C++, but I could settle for Objective-C or something else.
So far, I've experimented with:
1) DTrace scripting, which has taught me a lot, but the limitations of the D language (limited flow control, etc.) make it a major pain for what I'm doing, not to mention that the result would be a script, which isn't as tidy and self-contained as what I'm shooting for.
2) DYLD_INSERT_LIBRARIES interposition, which is slick in many ways, but perhaps due to namespace flattening (I won't pretend to deeply understand what this means), it works nicely against simpler executables, but makes my target application choke, even when I build a do-nothing library that doesn't actually interpose any calls.
My latest idea is to experiment with mach_star (https://github.com/rentzsch/mach_star), but I'm stopping here first, to ask the Stack Overflow community which invariably knows more than do I...
...should I be looking at something besides mach_star next?
I think you've made the right choice looking at mach_star.
If you actually want to learn how the darwin link-loader works, etc., I'd put more time into your DYLD insertion problems. But obviously you're looking for a quick solution, not an in-depth learning experience. And I doubt anyone's going to be able to figure out the problems you're having without having access to your project. So, this is probably a dead end. Besides, Mach overriding and injection are more fun anyway.
The basics of Mach injection aren't actually that hard, but there are a ton of things you have to get right, most of which aren't well documented. You're going to get 11 things wrong before you get something that works on your system, and then it won't work for the next function you try, and then it won't work on 10.5 or 10.8, and… The mach_star library wraps up all that stuff for you. So, why not use it?
I should mention that I haven't used mach_star since pre-Intel days. But it looks like it's still being updated regularly-ish, with changes for x86_64 and 10.7 and Xcode 4 and so on.