Sunday, January 2, 2011

The WebLogic Admin Server 11g just hangs upon startup

Problem:

Due to our disk space having filled up, I shutdown the Oracle WebLogic AdminServer 11g (11.1.1.3), cleared some logs, and restarted it.

However, when I start it up, it just stops at the entry below. No movement in the log whatsoever, and nothing is happening on the server.
oracle@oradev:/u01/app/oracle/middleware/user_projects/domains/soa_domain/bin> ./startWebLogic.sh
*****************************************************
** Setting up SOA specific environment...
*****************************************************
EXTRA_JAVA_PROPERTIES= -da:org.apache.xmlbeans...
.
LD_LIBRARY_PATH=/u01/app/oracle/middleware/patch_wls1033/profiles/default/native:/u01/app/oracle/middleware/Oracle_SOA1/lib:/u01/app/oracle/middleware/wlserver_10.3/server/native/linux/x86_64:/u01/app/oracle/middleware/wlserver_10.3/server/native/linux/x86_64/oci920_8:/u01/app/oracle/middleware/Oracle_SOA1/soa/thirdparty/edifecs/XEngine/bin
.
USER_MEM_ARGS=-Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
.
*****************************************************
** End SOA specific environment setup
*****************************************************
.
.
JAVA Memory arguments: -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
.
WLS Start Mode=Production
.
CLASSPATH=/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/user-patch.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/soa-startup.jar::/u01/app/oracle/middleware/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/jdk1.6.0_21/lib/tools.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules_10.3.3.0.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.common.adapters_11.1.1/oracle.soa.common.adapters.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/commons-cli-1.1.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.xdk_11.1.0/xsu12.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules.xquery_10.3.1.0.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/db2jcc4.jar:/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/soa-infra:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/fabric-url-handler_11.1.1.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/quartz-all-1.6.5.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.fabric_11.1.1/oracle.soa.fabric.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.adapter_11.1.1/oracle.soa.adapter.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.b2b_11.1.1/oracle.soa.b2b.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/app/oracle/middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/xqrl.jar:/u01/app/oracle/middleware/Oracle_SOA1/lib:/u01/app/oracle/middleware/jdk160_18/lib
.
PATH=/u01/app/oracle/middleware/wlserver_10.3/server/bin:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/bin:/u01/app/oracle/jdk1.6.0_21/jre/bin:/u01/app/oracle/jdk1.6.0_21/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:.
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Starting WLS with line:
/u01/app/oracle/jdk1.6.0_21/bin/java -server   -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true   -da -Dplatform.home=/u01/app/oracle/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/middleware/wlserver_10.3/server  -Ddomain.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dcommon.components.home=/u01/app/oracle/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig -Doracle.server.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/servers/AdminServer -Doracle.security.jps.config=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol  -Digf.arisidbeans.carmlloc=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/carml  -Digf.arisidstack.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.oamprovider_11.1.1  -Dweblogic.jdbc.remoteEnabled=true -Doracle.security.jps.policy.migration.validate.principal=false  -da:org.apache.xmlbeans... -Dsoa.archives.dir=/u01/app/oracle/middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dtangosol.coherence.clusteraddress=227.7.7.8 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dcom.sun.management.jmxremote -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/app/oracle/middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/app/oracle/middleware/oracle_common -Djava.awt.headless=true -Dbam.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dums.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/middleware/patch_wls1033/profiles/default/sysext_manifest_classpath  weblogic.Server
<Dec 22, 2010 2:46:25 PM EST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 17.0-b16 from Sun Microsystems Inc.>
<Dec 22, 2010 2:46:26 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.3.0  Fri Apr 9 00:05:28 PDT 2010 1321401 >
<Dec 22, 2010 2:46:30 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Dec 22, 2010 2:46:30 PM EST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Dec 22, 2010 2:46:30 PM EST> <Notice> <Log Management> <BEA-170019> <The server log file /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
Solution:

I worked an Oracle Support analyst (thanks Vijay!) and we reset some passwords and were able to bring up the server again. I am somewhat convinced that this was related to our disk being full.

This is what we proceeded doing:

1. Shutdown the Admin Server

2. Remove the "cache", "date", and "tmp" folders from the server AdminServer location
rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/cache 
rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data
rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/tmp
This did not solve our problem.

3. Reset the weblogic password by running this command:
java weblogic.security.utils.AdminAccount weblogic welcome1 .
... and resetting the password in:
./servers/AdminServer/security/boot.properties
This partially resolved the problem.

4. We noticed that the "system-jazn-data.xml" file was zero bytes in size:
ll /u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig
-rw-r----- 1 oracle ldba      0 Dec 21 20:49 system-jazn-data.xml
I copied the file from a separate server, and the Admin Server now started up.

The question remains... why was the "system-jazn-data.xml" file zero bytes?

Most likely when the disk was full, some process was probably trying to save to it but was unable to, thus blanking out the file. This is my only explanation for this.

Also, since the "system-jazn-data.xml" file that I copied over was not exactly identical, I had to recreate some user accounts.

Applicable Versions:
  • Oracle WebLogic Server 11g (10.3.3)
  • Oracle SOA Suite 11g (11.1.1.3)

3 comments:

Anonymous said...

Dear, I am having an issue that We are using Oracle 11g DB & Weblogic server ... when ever end user is connect with the Applications, v$session show the Weblogic Server Name and does not show the actual connecting Terminal as well as Operating System Name... There is any way to get the User Terminal as well as OS Name ???

Unknown said...

Hi,

I was facing the same issue other day.

Solution worked for me:
Uninstalling database. After uninstalling database restart server it will work without errors.

I don't know the reason why database is stopping weblogic.

Thanks,
Vijay

Arun said...

Hi
We are getting stuck after solaris patch process.
Truss shows stuck at keystore.xml and cwallet.sso as shown below

fstatat(AT_FDCWD, "/app/oracle_soa12c/u01/config/domains/soa_domain/config/fmwconfig/keystores.xml", 0xFFFFFFFF5B97E848, 0) = 0
8269/57: 1.290408
fstatat(AT_FDCWD, "/app/oracle_soa12c/u01/config/domains/soa_domain/config/fmwconfig/bootstrap/cwallet.sso", 0xFFFFFFFF5B97E848, 0) = 0

Any clues?
Keystore.xml and cwallet.sso are readable and have file size > 0