The WebLogic Admin Server 11g just hangs upon startup

Problem:

Due to our disk space having filled up, I shutdown the Oracle WebLogic AdminServer 11g (11.1.1.3), cleared some logs, and restarted it.

However, when I start it up, it just stops at the entry below. No movement in the log whatsoever, and nothing is happening on the server.

oracle@oradev:/u01/app/oracle/middleware/user_projects/domains/soa_domain/bin> ./startWebLogic.sh
*****************************************************
** Setting up SOA specific environment...
*****************************************************
EXTRA_JAVA_PROPERTIES= -da:org.apache.xmlbeans...
.
LD_LIBRARY_PATH=/u01/app/oracle/middleware/patch_wls1033/profiles/default/native:/u01/app/oracle/middleware/Oracle_SOA1/lib:/u01/app/oracle/middleware/wlserver_10.3/server/native/linux/x86_64:/u01/app/oracle/middleware/wlserver_10.3/server/native/linux/x86_64/oci920_8:/u01/app/oracle/middleware/Oracle_SOA1/soa/thirdparty/edifecs/XEngine/bin
.
USER_MEM_ARGS=-Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
.
*****************************************************
** End SOA specific environment setup
*****************************************************
.
.
JAVA Memory arguments: -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
.
WLS Start Mode=Production
.
CLASSPATH=/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/user-patch.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/soa-startup.jar::/u01/app/oracle/middleware/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/jdk1.6.0_21/lib/tools.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules_10.3.3.0.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.common.adapters_11.1.1/oracle.soa.common.adapters.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/commons-cli-1.1.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.xdk_11.1.0/xsu12.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules.xquery_10.3.1.0.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/db2jcc4.jar:/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/soa-infra:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/fabric-url-handler_11.1.1.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/quartz-all-1.6.5.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.fabric_11.1.1/oracle.soa.fabric.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.adapter_11.1.1/oracle.soa.adapter.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.b2b_11.1.1/oracle.soa.b2b.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/app/oracle/middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/xqrl.jar:/u01/app/oracle/middleware/Oracle_SOA1/lib:/u01/app/oracle/middleware/jdk160_18/lib
.
PATH=/u01/app/oracle/middleware/wlserver_10.3/server/bin:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/bin:/u01/app/oracle/jdk1.6.0_21/jre/bin:/u01/app/oracle/jdk1.6.0_21/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:.
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Starting WLS with line:
/u01/app/oracle/jdk1.6.0_21/bin/java -server   -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true   -da -Dplatform.home=/u01/app/oracle/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/middleware/wlserver_10.3/server  -Ddomain.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dcommon.components.home=/u01/app/oracle/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig -Doracle.server.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/servers/AdminServer -Doracle.security.jps.config=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol  -Digf.arisidbeans.carmlloc=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/carml  -Digf.arisidstack.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.oamprovider_11.1.1  -Dweblogic.jdbc.remoteEnabled=true -Doracle.security.jps.policy.migration.validate.principal=false  -da:org.apache.xmlbeans... -Dsoa.archives.dir=/u01/app/oracle/middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dtangosol.coherence.clusteraddress=227.7.7.8 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dcom.sun.management.jmxremote -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/app/oracle/middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/app/oracle/middleware/oracle_common -Djava.awt.headless=true -Dbam.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dums.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/middleware/patch_wls1033/profiles/default/sysext_manifest_classpath  weblogic.Server
<Dec 22, 2010 2:46:25 PM EST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 17.0-b16 from Sun Microsystems Inc.>
<Dec 22, 2010 2:46:26 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.3.0  Fri Apr 9 00:05:28 PDT 2010 1321401 >
<Dec 22, 2010 2:46:30 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Dec 22, 2010 2:46:30 PM EST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Dec 22, 2010 2:46:30 PM EST> <Notice> <Log Management> <BEA-170019> <The server log file /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>

Solution:

I worked an Oracle Support analyst (thanks Vijay!) and we reset some passwords and were able to bring up the server again. I am somewhat convinced that this was related to our disk being full.

This is what we proceeded doing:

1. Shutdown the Admin Server

2. Remove the "cache", "date", and "tmp" folders from the server AdminServer location

rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/cache 

rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data

rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/tmp

This did not solve our problem.

3. Reset the weblogic password by running this command:

java weblogic.security.utils.AdminAccount weblogic welcome1 .

... and resetting the password in:

./servers/AdminServer/security/boot.properties

This partially resolved the problem.

4. We noticed that the "system-jazn-data.xml" file was zero bytes in size:

ll /u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig

-rw-r----- 1 oracle ldba      0 Dec 21 20:49 system-jazn-data.xml

I copied the file from a separate server, and the Admin Server now started up.

The question remains... why was the "system-jazn-data.xml" file zero bytes?

Most likely when the disk was full, some process was probably trying to save to it but was unable to, thus blanking out the file. This is my only explanation for this.

Also, since the "system-jazn-data.xml" file that I copied over was not exactly identical, I had to recreate some user accounts.

Applicable Versions:

• Oracle WebLogic Server 11g (10.3.3)
• Oracle SOA Suite 11g (11.1.1.3)