Cannot start up WebLogic and NodeManager after Oracle Cloud data center outage

Problem

When starting Oracle WebLogic Server 11g AdminServer on an Oracle Cloud Infrastructure virtual machine, the WebLogic process was dying at startup. No explanation whatsoever as shown here:

INFO: Notification sent for Mapping config object reloaded 

<Jul 16, 2018, 8:20:01,49 PM EDT> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://adminhost.raastech.com:7002/jndi/weblogic.management.mbeanservers.editsession.DOMAIN.ServiceBus.> 

<Jul 16, 2018, 8:20:01,73 PM EDT> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://adminhost.raastech.com:7002/jndi/weblogic.management.mbeanservers.edit.> 

<Jul 16, 2018, 8:20:01,147 PM EDT> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://adminhost.raastech.com:7002/jndi/weblogic.management.mbeanservers.domainruntime.> 

Jul 16, 2018 8:20:01 PM oracle.ods.virtualization.engine.util.VDELogger info INFO: Notification sent for Mapping config object reloaded /u01/app/oracle/middleware/products/fmw1221/user_projects/domains/soa_domain/bin/startWebLogic.sh: line 203: 24671 Killed ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS} 

Stopping Derby server... 

Derby server stopped.

When starting up NodeManager, observed the following errors:

<Jul 16, 2018 8:21:11 PM EDT> <SEVERE> <Fatal error in NodeManager server>
java.io.IOException: The server is unable to create a server socket for listening. The address soahost1.raastech.com might be incorrect or another process is using port 5556: java.net.BindException: Cannot assign requested address
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:351)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:529)
        at weblogic.NodeManager.main(NodeManager.java:31)
Caused by: java.net.BindException: Cannot assign requested address
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:181)
        at sun.security.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:140)
        at sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:95)
        at weblogic.nodemanager.server.SSLListener.privilegedInit(SSLListener.java:134)
        at weblogic.nodemanager.server.SSLListener.access$000(SSLListener.java:33)
        at weblogic.nodemanager.server.SSLListener$1.run(SSLListener.java:97)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLListener.init(SSLListener.java:94)
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:349)
        ... 2 more

Failed attempts at resolution:

• Started WebLogic manually. No help.
• Increased debugging. No additional information.
• Reset the 'weblogic' password. No help.
• Cleared the ~/tmp ~/cache ~/data folders of the AdminServer. No help.
• Restored from backup. No help.
• This behavior appeared to be attributed to memory issues, so shut down every single service on the box. No help.

Note that there was an outage at the Oracle Cloud data center, which killed our VM. All problems started happening after the VM was started up.

Solution

1. Memory hardware issue. Rebuild the VM.

For example:

a. Stop the VM.
b. Create a new VM.
c. Mount the storage volumes on the new VM.
d. Rename the hostname of the new VM.
e. Everything will start up smoothly with no problems.

Applicable Versions

• Oracle WebLogic Server (10.3.6)
• Oracle Cloud Infrastructure (2019)