perl + DBD::Oracle + mod_perl + oracle LDAP name resolver = crash?

Question

This happened to me on an older Fedora system. Today, i set up a new CentOS 6.5, installed the newest oracle client (12.1.0.1.0), the newest DBD::Oracle from cpan (1.68), and had the same problem: apache segfaults when i try to connect to a database. When i try the same perl from the command line, or as CGI, it works; when i turn off the LDAP name resolver in the sqlnet.ora file, it works as well.

To start from the beginning: This is my startup.pl:

$ENV{'NLS_LANG'}='AMERICAN_AMERICA.AL32UTF8';
$ENV{'ORACLE_HOME'}='/opt/oracle/OraHome1';
use DBI ();
use DBD::Oracle ();

and this is my test program:

#!/usr/bin/perl
require "startup.pl" unless defined $ENV{'ORACLE_HOME'};
use DBI;
use DBD::Oracle;
print "Content-type: text/plain\n\n";
print "connecting: \n";
my $dbh=DBI->connect("dbi:Oracle:mydb", 'username', 'password');
print "connected! \n";

Running this from the shell works well. Running as CGI script works well. Running it from mod_perl crashes apache:

[Wed Dec 04 16:38:01 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Wed Dec 04 16:38:28 2013] [notice] child pid 25756 exit signal Segmentation fault (11)

Doing an strace on the httpd process shows the following:

connect(21, {sa_family=AF_INET, sin_port=htons(389), sin_addr=inet_addr("10.250.52.237")}, 16) = 0
write(21, "0\f\2\1\1`\7\2\1\2\4\0\200\0", 14) = 14
poll([{fd=21, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, -1) = 1 ([{fd=21, revents=POLLIN}])
read(21, "0\204\0\0\0\20\2\1", 8)       = 8
read(21, "\1a\204\0\0\0\7\n\1\0\4\0\4\0", 14) = 14
write(21, "0\201\226\2\1\2c\201\220\0042cn=bpas_p,cn=OracleCo"..., 153) = 153
poll([{fd=21, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=21, revents=POLLIN}])
read(21, "0\204\0\0\1{\2\1", 8)         = 8
read(21, "\2d\204\0\0\1r\0040cn=bpas_p,cn=OracleCont"..., 377) = 377
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

So, obviously, the httpd crashes while talking to 10.250.52.237:389, which is the ldap port of the oracle names resolver.

My standard sqlnet.ora is:

NAMES.DIRECTORY_PATH    = (LDAP, ONAMES, TNSNAMES)
NAMES.DEFAULT_DOMAIN    = xxx.yyyy.zzz
NAMES.PREFERRED_SERVERS =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oranames01.yyyy.zzz)(PORT = 1501))
    (ADDRESS = (PROTOCOL = TCP)(HOST = oranames02.yyyy.zzz)(PORT = 1502))
  )

and this is my ldap.ora:

DIRECTORY_SERVERS = (oranames01.yyyy.zzz:389:636, oranames02.yyyy.zzz:389:636)
DEFAULT_ADMIN_CONTEXT = "dc=xxx, dc=yyyy, dc=zzz"
DIRECTORY_SERVER_TYPE = OID

The address from the strace dump (10.250.52.237) is the IP address of oranames01.yyyy.zzz.

Now, if i remove the LDAP adapter from sqlnet.ora

NAMES.DIRECTORY_PATH    = (ONAMES, TNSNAMES)

and put the database connect string into tnsnames.ora

mydb.xxx.yyyy.zzz=(DESCRIPTION=(SOURCE_ROUTE=OFF)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ivorapo01.xxx.yyyy.zzz)(PORT=15350)))(CONNECT_DATA=(SERVICE_NAME=mydb.xxx.yyyy.zzz)(FAILOVER_MODE =(TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 1))))

everything works well - command line, cgi perl and mod_perl.

So, obviously, there seems to be some problem involving the combination of mod_perl and the LDAP name resolver.

This happened to me on a Fedora 14 system with apache 2.2.17 and Oracle client 10.2.0.1.0, first, and with a freshly installed CentOs 6.5, apache 2.2.15, and oracle client 12.1.0.1.0 today. Google'ing for mod_perl and oracle shows some hits refering to ORACLE_HOME being set too late (i had this as well, putting it into startup.pl fixes that problem), but nothing that seems to fit my problem.

Can anyone confirm this? Or, can anyone confirm mod_perl + DBD::Oracle + oracle LDAP name resolving working on their site? Right now, i can work around the problem by not using the LDAP names adapter, but i'd rather like to fix that before going in production.

Answer 1

I traced this to the ldap library, /lib64/libldap-2.4.so.2. This library is used by mod_ldap, mod_authnz_ldap, and mod_php through an optional dependency (It seems that the dependency list is php -> libcurl -> libssl -> libldap, but i didn't really investigate). This library duplicates several function names that oracle defines in libclntsh.so as well. Obviously, the functions aren't compatible with each other. In my case, the crash was in ldap_search_st.

Once i removed mod_php, mod_ldap and mod_authnz_ldap from my server, the crashes stopped.

Since the dependency of php from libldap seems to be optional, i tried re-enabling php, and renaming /lib64/libldap-2.4.so.2 to /lib64/libldap-2.4.so.2.disabled so php wouldn't load it. I re-enabled php, and the server still didn't crash.

I don't need php on my server, so i disabled php again and undid the library rename. But it seems the solution is:

a) You can't use the oracle LDAP name resolver together with the system LDAP library in one process.

b) If you want to use the LDAP resolver, you need to disable mod_ldap and mod_authnz_ldap.

c) If your server doesn't need php, disable it as well. If it does need php, you can leave it enabled, but you have to make sure it doesn't load the system ldap library. Ldap functions in php, as well as curl which depends on it, will not work.

d) You can rename the system ldap library to make sure it doesn't get pulled in, but this will kill everything else on your system that depends on LDAP. (Note that the Net::LDAP module in perl does NOT use this library so it will still work after the rename!).

e) If you really NEED php and NEED the ldap library, the only workaround i see is copying libldap-2.4.so.2 to some directory that isn't in /etc/ld.so.cache, and set LD_LIBRARY_PATH to that directory for the programs that need the library (but not for the http server, obviously).