RUST error ENOMEM in call of posix_memalign into attached program

Question

I'll try to summarize but it's gonna be complicated.

I'm having an operating system course in my university, and i have a lab work to do. I'm working in Rust (the lab work is said to be doable in any compiled language but was principally designed for C in the first time).

So I have a tracer program and a traced program.

The goal of this step of the lab work is to attach from the tracer to the traced with ptrace, then to inject "trap-call-trap" instructions at the good place to replace an existing useless function, with the function posix_memalign (from the libc) in the traced, by an indirect call via registers (with rax). The goal is to allocate memory to be able later to call a cache code from another file into the traced program.

The problem i have is that i achieve to do a posix memalign in the tracer so i know the function works, but when i call it in the traced (via the tracer), and look in register rax for the return of the function, i always get a "12" which corresponds to ENOMEM (ENOMEM Not enough space/cannot allocate memory).

I have 2 separated cargo projects to be able to launch each program separately from cargo. Everything is on this git : https://github.com/Carrybooo/TP_SEL

I'm sorry all prints and outputs are in french, (some comments too, i'm in a french course) and i didn't think i would have to share it with anyone. There is a lot of useless code too in it, from the previous steps of the lab, that i have kept just in case, so my code is not really clean.

This is the part where i attach and modify the regs to call the function (I shortened the code and didn't show you the auxiliaries functions declarations cause it would be too long) :

    ptrace::attach(pid_ptrace) //attaching to process

    wait(); //wait after attaching

    inject(pid_trace, offset_fct_to_replace, false); //injecting trap-call-trap
    ptrace::cont(pid_ptrace, Signal::SIGCONT);

    wait(); //wait for 1st trap

    let mut regs =
        ptrace::getregs(pid_ptrace);

    let ptr_to_ptr: *mut *mut c_void = ptr::null_mut();

    regs.rax = get_libc_address(pid_trace).unwrap() + get_libc_offset("posix_memalign").unwrap();
    regs.rsp = regs.rsp - (size_of::<*mut *mut c_void>() as u64);
    regs.rdi = ptr_to_ptr as u64;
    regs.rsi = size_of::<usize>() as u64;
    regs.rdx = size_of::<usize>() as u64;

    ptrace::setregs(pid_ptrace, regs); //set regs with modification

    ptrace::cont(pid_ptrace, Signal::SIGCONT);

    wait();

    let regs = ptrace::getregs(pid_ptrace);

    ptrace::detach(pid_ptrace, Signal::SIGCONT) //detaching

And running the program in the terminal gives something like this :

before modification of regs:
rax = 6
rip = 55e6c932ddc1
rsp = 7ffcee0b7fb8

before function execution:
rax = 7f59b935ded0
rdi = 0
rip = 55e6c932ddc1
rsp = 7ffcee0b7fb0

after function execution:
rax = 12                                   <------//RESULT OF THE CALL IS HERE
rdi = 55e6cac6aba0
rip = 7f59b935df20
rsp = 7ffcee0b7f90

//end of program

So yeah i don't know why i keep getting an error on this call, i presume it's because it violates rust memory safety, because the compiler never knew in the traced program that it would have to allocate memory, but i'm not sure of it, neither how to bypass it. I hope that i've been clear enough, let me know if you need any more detail, and i really thank in advance anyone who could help me. Every advice will be welcome.

Answer 1

So I went with the libc memalign instead of using posix_memalign. It's not as restricted, and it works well while calling with registers in the traced program.

Still no idea of why posix_memalign doesn't work while called from the regs in a rust program.