Cannot resolve reference to bean 'DOStoreFactory' during SOA startup

Problem

When starting up "soa_server1" it comes up fine. But when we start "soa_server2", we get these errors in the soa_server2.out log file:

<Mar 2, 2018 1:39:56 PM GMT> <Warning> <HTTP> <BEA-101162> <User defined listener com.oracle.weblogic.wsee.wrapper.org.springframework.web.context.ContextLoaderListener failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SensorManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricMesh' while setting bean property 'fabricMesh'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMesh': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMeshTarget' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'WSServiceBC' while setting bean property 'bindingComponents' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'WSServiceBC' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'InstanceManager' while setting bean property 'instanceManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'InstanceManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'DOStoreFactory' while setting bean property 'DOStoreFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DOStoreFactory' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricConfigManager' while setting bean property 'fabricConfigManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricConfigManager' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'EventBusConfigurable' while setting bean property 'configurables' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'EventBus' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'ClusterInterface' while setting bean property 'clusterInterface'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ClusterInterface' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
 MemberSet=MasterMemberSet(
 ThisMember=Member(Id=2, Timestamp=2018-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
 OldestMember=Member(Id=1, Timestamp=2018-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
 ActualMemberSet=MemberSet(Size=2
 Member(Id=1, Timestamp=2018-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
 Member(Id=2, Timestamp=2018-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
 )
 MemberId|ServiceVersion|ServiceJoined|MemberState
 1|3.7.1|2018-05-02 13:34:29.559|JOINING,
 2|3.7.1|2018-05-02 13:34:56.523|JOINED
 RecycleMillis=1200000
 RecycleSet=MemberSet(Size=0
 )
 )
 ).

You may also see this error:

<Mar 2, 2018 1:41:26 PM GMT> <Error> <Cluster> <BEA-000140> <Failed to deserialize statedump from server 6011237482029920459S:192.168.0.21:[8001,8001,-1,-1,-1,-1,-1]:soahost1:8001,soahost2:8001:soadomain:soa_server2 with java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote.
 java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote
 at weblogic.application.internal.AppClassLoaderManagerImpl.loadApplicationClass(AppClassLoaderManagerImpl.java:164)
 at weblogic.common.internal.ProxyClassResolver.resolveProxyClass(ProxyClassResolver.java:68)
 at weblogic.common.internal.WLObjectInputStream.resolveProxyClass(WLObjectInputStream.java:76)
 at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
 Truncated. see log file for complete stacktrace
 >

Even if you have opened up all relevant ports, the problem persists:

root@soahost1> iptables -S | grep IN_public_allow
 -N IN_public_allow
 -A IN_public -j IN_public_allow
 -A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 7001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 8001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 9001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 5556 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 7 -m conntrack --ctstate NEW -j ACCEPT

Solution

1. Turn off the local firewall:

service firewalld stop

You may need to spend more time to open up UDP and multicast on every NIC to re-enable the local firewall.

References

• https://docs.oracle.com/middleware/1213/coherence/tutorial/installcoh.htm#COHTU112 

Applicable Versions

• Oracle SOA Suite 11g (11.1.1.9.0)