I hooked system calls open, read, write, lstat etc. using osxinj project. Injected this into TextEdit application provided by apple and everything worked fine. When I opened new file using textedit, opencallback was called and messages were logged in system.log file.
typedef int (*open_type)(const char *, int, mode_t);
open_type open_func = 0;
int opencallback(const char* path, int oflag, mode_t mode)
{
syslog(LOG_ALERT, "In open...");
int returnVal = open_func(path, oflag, mode);
syslog(LOG_ALERT,"Open, ends\n");
return returnVal;
}
Injected into Excel and tried to override open system call using below code:
void* func_ptr = dlsym( RTLD_NEXT, "open");
if (func_ptr)
{
open_func = (open_type)func_ptr;
mach_error_t me = mach_override_ptr( func_ptr,
(void*)&opencallback,
(void**)&open_func);
}
opencallback is called when injecting to TextEdit but it is not getting called when injected in Microsoft Excel. But code written on same lines for other system calls read, write, lstat are getting interposed when injected in Excel.
Any thoughts on why open is not getting interposed when injected to Excel.
Finally, I got my code running. I am posting answer hoping it might help somebody. I hooked __open which is an alias for open and it worked fine for i386 application like excel.