Java 1.6.0_45 on Solaris 9 returns relocation error "symbol __fmodf: referenced symbol not found"

Question

I have a java application running in a Solaris 9 (brand zone inside a Solaris 10 global).

root@server # cat /etc/release            
                        Solaris 9 4/03 s9s_u3wos_08 SPARC
           Copyright 2003 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 25 February 2003

...

root@server # isainfo -v 
64-bit sparcv9 applications
32-bit sparc applications

I know for sure (from the logs) that the application used to run for months on top of a JDK 1.6.0_45:

root@server # pkginfo | grep -i jdk                    
system      SUNWj2dem            JDK 1.2 demo programs
system      SUNWj2man            JDK 1.2 man pages
system      SUNWj2rt             JDK 1.2 run time environment
system      SUNWj3irt            JDK 1.4 I18N run time environment
system      SUNWj6cfg            JDK 6.0 Host Config. (1.6.0_45)
system      SUNWj6dev            JDK 6.0 Dev. Tools (1.6.0_45)
system      SUNWj6dvx            JDK 6.0 64-bit Dev. Tools (1.6.0_45)
system      SUNWj6jmp            JDK 6.0 Man Pages: Japan (1.6.0_45)
system      SUNWj6man            JDK 6.0 Man Pages (1.6.0_45)
system      SUNWj6rt             JDK 6.0 Runtime Env. (1.6.0_45)
system      SUNWj6rtx            JDK 6.0 64-bit Runtime Env. (1.6.0_45) 

Now, after a reboot, Java 6 returns an error, while the other versions still run fine:

root@server # java -version
dl failure on line 685Error: failed /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/libjvm.so, because ld.so.1: java: fatal: relocation error: file /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/libjvm.so: symbol __fmodf: referenced symbol not found

I know this is an expected behavior in Solaris 8 and/or with JDK 7, but Oracle certifies JDK 6 as Solaris 9 compatible, so I can't really figure out what's happening. I've been stuck on this for two days and tried possibly any workaround but no luck.

All the needed system libraries are resolved:

root@server # ldd -v /usr/bin/java                                                                

   find object=/usr/lib/secure/s9_preload.so.1; required by /usr/jdk/instances/jdk1.6.0/bin/java
        /usr/lib/secure/s9_preload.so.1

   find object=libthread.so.1; required by /usr/jdk/instances/jdk1.6.0/bin/java
        libthread.so.1 =>        /usr/lib/libthread.so.1
   find version=libthread.so.1
        libthread.so.1 (SISCD_2.3a) =>   /usr/lib/libthread.so.1

   find object=libjli.so; required by /usr/jdk/instances/jdk1.6.0/bin/java
        libjli.so =>     /usr/jdk/instances/jdk1.6.0/bin/../jre/lib/sparc/jli/libjli.so
   find version=libjli.so
        libjli.so (SUNWprivate_1.1) =>   /usr/jdk/instances/jdk1.6.0/bin/../jre/lib/sparc/jli/libjli.so

   find object=libdl.so.1; required by /usr/jdk/instances/jdk1.6.0/bin/java
        libdl.so.1 =>    /usr/lib/libdl.so.1
   find version=libdl.so.1
        libdl.so.1 (SUNW_0.8) =>         /usr/lib/libdl.so.1

   find object=libc.so.1; required by /usr/jdk/instances/jdk1.6.0/bin/java
        libc.so.1 =>     /usr/lib/libc.so.1
   find version=libc.so.1
        libc.so.1 (SUNW_0.7) =>  /usr/lib/libc.so.1
        libc.so.1 (SUNWprivate_1.1) =>   /usr/lib/libc.so.1

   find object=libc.so.1; required by /usr/lib/secure/s9_preload.so.1

   find object=libc.so.1; required by /usr/lib/libthread.so.1
   find version=libc.so.1
        libc.so.1 (SUNW_1.21.2) =>       /usr/lib/libc.so.1
        libc.so.1 (SUNWprivate_1.1) =>   /usr/lib/libc.so.1

   find object=libdl.so.1; required by /usr/lib/libthread.so.1
   find version=libdl.so.1
        libdl.so.1 (SUNWprivate_1.1) =>  /usr/lib/libdl.so.1

   find object=libc.so.1; required by /usr/jdk/instances/jdk1.6.0/bin/../jre/lib/sparc/jli/libjli.so
   find version=libc.so.1
        libc.so.1 (SUNW_0.7) =>  /usr/lib/libc.so.1

   find object=libdl.so.1; required by /usr/lib/libc.so.1
   find version=libdl.so.1
        libdl.so.1 (SUNW_1.4) =>         /usr/lib/libdl.so.1
        libdl.so.1 (SUNWprivate_1.1) =>  /usr/lib/libdl.so.1

   object=/usr/lib/libdl.so.1; filter for /usr/lib/ld.so.1

   object=/usr/lib/libc.so.1; filter for /usr/platform/$PLATFORM/lib/libc_psr.so.1

   find object=/usr/platform/SUNW,Sun-Blade-T6320/lib/libc_psr.so.1; required by /usr/lib/libc.so.1
        /usr/platform/SUNW,Sun-Blade-T6320/lib/libc_psr.so.1

So I must assume that for some reason they are outdated and do not implement the symbol __fmodf. But how it worked before?

As far as I can understand, fmodf() is a method from the libm library. Since Solaris 9 links libm.so to libm.so.1 by default, I also tried to override the math libraries with

root@server # LD_PRELOAD=/.SUNWnative/lib/libm.so.2 java -version 

But still no luck.

Is there any way to override the symbol and/or to run Java in some sort of "compatibility" mode to fix the issue?

Or am I just missing something very obvious?

Thanks

EDIT: as suggested by Andrew Henle, this is the complete output of ldd for libjvm.so:

root@server # ldd -rv /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/libjvm.so

   find object=libc.so.1; required by /usr/lib/secure/s9_preload.so.1
        libc.so.1 =>     /usr/lib/libc.so.1

   find object=libdl.so.1; required by /usr/lib/libc.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
   find version=libdl.so.1
        libdl.so.1 (SUNW_1.4) =>         /usr/lib/libdl.so.1
        libdl.so.1 (SUNWprivate_1.1) =>  /usr/lib/libdl.so.1

   object=/usr/lib/libc.so.1; filter for /usr/platform/$PLATFORM/lib/libc_psr.so.1

   find object=/usr/platform/SUNW,Sun-Blade-T6320/lib/libc_psr.so.1; required by /usr/lib/libc.so.1
        /usr/platform/SUNW,Sun-Blade-T6320/lib/libc_psr.so.1

   object=/usr/lib/libdl.so.1; filter for /usr/lib/ld.so.1

...

root@server # ldd -ss /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/libjvm.so

   find object=libc.so.1; required by /usr/lib/secure/s9_preload.so.1
    search path=/usr/openwin/lib:/usr/local/lib:/usr/local/ssl/lib  (LD_LIBRARY_PATH)
    trying path=/usr/openwin/lib/libc.so.1
    trying path=/usr/local/lib/libc.so.1
    trying path=/usr/local/ssl/lib/libc.so.1
    search path=/usr/lib  (default)
    trying path=/usr/lib/libc.so.1
        libc.so.1 =>     /usr/lib/libc.so.1

   find object=libdl.so.1; required by /usr/lib/libc.so.1
    search path=/usr/openwin/lib:/usr/local/lib:/usr/local/ssl/lib  (LD_LIBRARY_PATH)
    trying path=/usr/openwin/lib/libdl.so.1
    trying path=/usr/local/lib/libdl.so.1
    trying path=/usr/local/ssl/lib/libdl.so.1
    search path=/usr/lib  (default)
    trying path=/usr/lib/libdl.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1

   object=/usr/lib/libc.so.1; filter for /usr/platform/$PLATFORM/lib/libc_psr.so.1

   find object=/usr/platform/SUNW,Sun-Blade-T6320/lib/libc_psr.so.1; required by /usr/lib/libc.so.1
        /usr/platform/SUNW,Sun-Blade-T6320/lib/libc_psr.so.1

   object=/usr/lib/libdl.so.1; filter for /usr/lib/ld.so.1

Answer 1

Ok, solved. This is quite weird, though.

As suggested by Andrew Henle in the comments, I examined carefully the output of

root@server # truss -f -a -e -l -f -rall -wall -o truss_jdk_1.6.0_45.txt /usr/bin/java -version

And confirmed that the math library containing the symbol __fmodf is called at runtime by libjvm.so, that look for it in several places before finding it in the system's default location:

24757/1:        stat("/usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/libm.so.1", 0xFFBFE910) Err#2 ENOENT
24757/1:        stat("/usr/jdk/instances/jdk1.6.0/jre/lib/sparc/libm.so.1", 0xFFBFE910) Err#2 ENOENT
24757/1:        stat("/usr/jdk/instances/jdk1.6.0/jre/../lib/sparc/libm.so.1", 0xFFBFE910) Err#2 ENOENT
24757/1:        stat("/usr/openwin/lib/libm.so.1", 0xFFBFE910)  Err#2 ENOENT
24757/1:        stat("/usr/local/lib/libm.so.1", 0xFFBFE910)    Err#2 ENOENT
24757/1:        stat("/usr/local/ssl/lib/libm.so.1", 0xFFBFE910) Err#2 ENOENT
24757/1:        stat("/usr/lib/libm.so.1", 0xFFBFE910)          = 0
24757/1:        resolvepath("/usr/lib/libm.so.1", "/usr/lib/libm.so.1", 1023) = 18
24757/1:        open("/usr/lib/libm.so.1", O_RDONLY)            = 3

The file /usr/lib/libm.so.1 actually exists on the system, but I had also noticed before that other versions of the same math library are installed elsewhere:

/.SUNWnative/lib/libm.so.1
/.SUNWnative/lib/libm.so.2

So I tried to feed Java with those, by soft-linking them into /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/.

When I linked /.SUNWnative/lib/libm.so.1 nothing changed, but then I tried the dirtiest trick out of desperation:

root@server # ln -s /.SUNWnative/lib/libm.so.2 /usr/jdk/instances/jdk1.6.0/jre/lib/sparc/server/libm.so.1

And this, surprisingly, did the trick. Now Java 6 update 45 runs smoothly on my Solaris 9.

So, libjvm.so explicitly looks for libm.so.1 (not just libm.so, that is usually a symlink to the default version of the actual library) but it actually needs libm.so.2 to work...

Thanks a lot to Andrew for the hints!