FAILED_NOT_RESTARTABLE in WebLogic Server 11g

Problem:

In WebLogic Server 11g, you suddently get a FAILED_NOT_RESTARTABLE state.

Solution:

1. Check the last few entries in the output file in the logs directory for the root cause (e.g., $DOMAIN_HOME/servers/soa_server1/logs/soa_server1.out).

In this example, the error was obvious as it was due to the file system running out of disk space:

<Aug 13, 2012 10:31:26 AM> <SEVERE> <NodeManager> <Unexpected error while monitoring server>
java.io.IOException: Disk quota exceeded
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.checkAndCreate(File.java:1704)
        at java.io.File.createTempFile(File.java:1792)
        at weblogic.nodemanager.util.ConcurrentUnixFile.write(ConcurrentUnixFile.java:38)
        at weblogic.nodemanager.common.StateInfoWriter.writeStateInfo(StateInfoWriter.java:45)
        at weblogic.nodemanager.common.StateInfo.save(StateInfo.java:152)
        at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess(AbstractServerMonitor.java:358)
        at weblogic.nodemanager.server.AbstractServerMonitor.runMonitor(AbstractServerMonitor.java:333)
        at weblogic.nodemanager.server.AbstractServerMonitor.run(AbstractServerMonitor.java:241)
        at weblogic.nodemanager.server.ServerMonitor.run(ServerMonitor.java:25)
        at java.lang.Thread.run(Thread.java:662)

Applicable Versions:

• Oracle WebLogic Server 11g (11.1.1.5)