Java Foreign API: how to pass pointers and pointers address?

Question

I am working with the new Java Foreign API and I have to call two C functions (Java bindings generated using the JExtract tool) that take in input a double pointer and a pointer:

• https://yara.readthedocs.io/en/stable/capi.html#c.yr_compiler_create
• https://yara.readthedocs.io/en/stable/capi.html#c.yr_compiler_destroy

---

int yr_compiler_create(YR_COMPILER** compiler)

void yr_compiler_destroy(YR_COMPILER* compiler)

---

In the official YARA C API tests these methods are called as follows:

#include <yara.h>

voit test_compiler(){
    YR_COMPILER* compiler = NULL;
    yr_initialize();

    yr_compiler_create(&compiler);
    yr_compiler_destroy(compiler);

    yr_finalize();
}

---

The corresponding Java bindings are as follows:

/**
 * {@snippet :
 * int yr_compiler_create(YR_COMPILER** compiler);
 * }
 */
public static int yr_compiler_create(MemorySegment compiler) {
    var mh$ = yr_compiler_create$MH();
    try {
        return (int)mh$.invokeExact(compiler);
    } catch (Throwable ex$) {
        throw new AssertionError("should not reach here", ex$);
    }
}

/**
 * {@snippet :
 * void yr_compiler_destroy(YR_COMPILER* compiler);
 * }
 */
public static void yr_compiler_destroy(MemorySegment compiler) {
    var mh$ = yr_compiler_destroy$MH();
    try {
        mh$.invokeExact(compiler);
    } catch (Throwable ex$) {
        throw new AssertionError("should not reach here", ex$);
    }
}

All the generated bindings for the yara.h header file are available at https://github.com/YARA-Java/YARA-Java/tree/master/yara-java

Going to the actual issue, I'm calling these methods from Java as follows:

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;

import static com.virustotal.yara.yara_h.ERROR_SUCCESS;
import static com.virustotal.yara.yara_h_1.*;

public class YaraTest {
    @Test
    public void testCompiler(){
        yr_initialize();

        try (Arena arena = Arena.openConfined()) {
            MemorySegment compiler = MemorySegment.allocateNative(YR_COMPILER.$LAYOUT(), arena.scope());

            int created = yr_compiler_create(compiler);
            Assertions.assertEquals(ERROR_SUCCESS(), created);

            yr_compiler_destroy(compiler);
        }

        yr_finalize();
    }
} 

but it results in the following error, when calling the yr_compiler_destroy method:

double free or corruption (out)

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

I did also try to create a C_POINTER (represented by a generated layout class),

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;

import static com.virustotal.yara.yara_h.ERROR_SUCCESS;
import static com.virustotal.yara.yara_h_1.*;

public class YaraTest {
    @Test
    public void testCompiler(){
        yr_initialize();

        try (Arena arena = Arena.openConfined()) {
            MemorySegment compiler = MemorySegment.allocateNative(YR_COMPILER.$LAYOUT(), arena.scope());
            MemorySegment compilerAddress = MemorySegment.allocateNative(Constants$root.C_POINTER$LAYOUT, arena.scope());
            compilerAddress.set(Constants$root.C_POINTER$LAYOUT, 0, MemorySegment.ofAddress(compiler.address()));

            int created = yr_compiler_create(compilerAddress);
            Assertions.assertEquals(ERROR_SUCCESS(), created);

            yr_compiler_destroy(compiler);
        }

        yr_finalize();
    }
}

but it results in a fatal error:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9ae98dab44, pid=75962, tid=75963
#
# JRE version: OpenJDK Runtime Environment (20.0+29) (build 20-ea+29-2280)
# Java VM: OpenJDK 64-Bit Server VM (20-ea+29-2280, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libyara.so.8.0.0+0x3cb44]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /mnt/bytes/Workspace/YARA-Java/yara-java/yara-java/core.75962)
#
# An error report file with more information is saved as:
# /mnt/bytes/Workspace/YARA-Java/yara-java/yara-java/hs_err_pid75962.log
[0,501s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

The full log is available here https://pastebin.com/a6xp6rLD.

What is the equivalent of YR_COMPILER** and YR_COMPILER* using Java Foreign API?

Answer 1

Code using the & operator in C can generally not be directly translated into Java. Instead you have to allocate the type passed to the & operator directly.

YR_COMPILER* compiler = NULL;
yr_compiler_create(&compiler);

Here the type of compiler is YR_COMPILER* so the layout is C_POINTER.

Your second attempt is really close, but you also have to read the pointer back from the compilerAddress segment. What your code does is this:

compiler -> YR_COMPILER (uninitialized)
compilerAddress -> C_POINTER (value is address of 'compiler')

Then the function call happens, which changes the value pointed to by compilerAddress, and you get:

compiler -> YR_COMPILER (uninitialized)
compilerAddress -> C_POINTER (value is address set by 'yr_compiler_create')

Since you then pass the uninitialized compiler pointer to yr_compiler_destroy it is not surprising the program crashes.

So, you just have to read back the pointer from compilerAddress after calling yr_compiler_create:

try (Arena arena = Arena.openConfined()) {
    MemorySegment compilerAddress = arena.allocate(C_POINTER); // YR_COMPILER**

    int created = yr_compiler_create(compilerAddress);
    Assertions.assertEquals(ERROR_SUCCESS(), created);

    MemorySegment compiler = compilerAddress.get(C_POINTER, 0); // YR_COMPILER*

    yr_compiler_destroy(compiler);
}

(Note that the C_POINTER layout should be generated in yara_h.java)

Essentially, the C code equivalent would be:

YR_COMPILER** compilerAddress = malloc(sizeof *compilerAddress);
yr_compiler_create(compilerAddress);
YR_COMPILER* compiler = *compilerAddress;

Which doesn't use &.