BEA-171520 and "Could not obtain an exclusive lock for directory" when starting up WebLogic 11g AdminServer

Problem:

If you have configured your Oracle WebLogic 11g server to perform LDAP authenticaion, when starting up the WebLogic 11g AdminServer, you may receive the following error:

<Dec 3, 2010 8:58:34 AM EST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>

Investigation:

Upon investigating the AdminServer log file, located here:

/u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/logs/AdminServer.log

You may notice the following, which pretty much indicates the same thing:

####<Dec 3, 2010 8:58:54 AM EST> <Warning> <EmbeddedLDAP> <oradev> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1291384734208> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>

After shutting down the AdminServer, do a 'ps' for the remaining running processes, and you may find lingering processes which may have to manually be killed.

oracle@oradev:/home/oracle> ps -ef | grep soa | grep -v grep
oracle   29085 24648  0 09:32 pts/3    00:00:00 /bin/sh /u01/app/oracle/middleware/user_projects/domains/soa_domain/bin/startWebLogic.sh
oracle   29133 29085 45 09:32 pts/3    00:00:37 /u01/app/oracle/jdk1.6.0_21/bin/java -server -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/u01/app/oracle/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/middleware/wlserver_10.3/server -Ddomain.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dcommon.components.home=/u01/app/oracle/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig -Doracle.server.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/servers/AdminServer -Doracle.security.jps.config=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=true -Doracle.security.jps.policy.migration.validate.principal=false -da:org.apache.xmlbeans... -Dsoa.archives.dir=/u01/app/oracle/middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dtangosol.coherence.clusteraddress=227.7.7.8 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dcom.sun.management.jmxremote -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/app/oracle/middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/app/oracle/middleware/oracle_common -Djava.awt.headless=true -Dbam.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dums.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/middleware/patch_wls1033/profiles/default/sysext_manifest_classpath weblogic.Server

Solution:

1. Shutdown the AdminServer

2. Do a "ps -ef | grep oracle | grep -v grep" and observe and see if you have any lingering processes

3. If so, then do a "kill <processid>"

4. If that doesn't kill the process gracefully, then do a "kill -9 <processid>"

5. Restart the AdminServer and you should be okay

Applicable Versions:

• Oracle WebLogic 11g (11.1.1.x)