Raastech Blog: Cannot resolve reference to bean 'DOStoreFactory' during SOA startup

Problem

When starting up "soa_server1" it comes up fine. But when we start "soa_server2", we get these errors in the soa_server2.out log file:

<Mar 2, 2018 1:39:56 PM GMT> <Warning> <HTTP> <BEA-101162> <User defined listener com.oracle.weblogic.wsee.wrapper.org.springframework.web.context.ContextLoaderListener failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SensorManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricMesh' while setting bean property 'fabricMesh'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMesh': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMeshTarget' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'WSServiceBC' while setting bean property 'bindingComponents' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'WSServiceBC' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'InstanceManager' while setting bean property 'instanceManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'InstanceManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'DOStoreFactory' while setting bean property 'DOStoreFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DOStoreFactory' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricConfigManager' while setting bean property 'fabricConfigManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricConfigManager' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'EventBusConfigurable' while setting bean property 'configurables' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'EventBus' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'ClusterInterface' while setting bean property 'clusterInterface'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ClusterInterface' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=Member(Id=2, Timestamp=2018-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
OldestMember=Member(Id=1, Timestamp=2018-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
ActualMemberSet=MemberSet(Size=2
Member(Id=1, Timestamp=2018-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
Member(Id=2, Timestamp=2018-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2018-05-02 13:34:29.559|JOINING,
2|3.7.1|2018-05-02 13:34:56.523|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
).

You may also see this error:

<Mar 2, 2018 1:41:26 PM GMT> <Error> <Cluster> <BEA-000140> <Failed to deserialize statedump from server 6011237482029920459S:192.168.0.21:[8001,8001,-1,-1,-1,-1,-1]:soahost1:8001,soahost2:8001:soadomain:soa_server2 with java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote.
java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote
at weblogic.application.internal.AppClassLoaderManagerImpl.loadApplicationClass(AppClassLoaderManagerImpl.java:164)
at weblogic.common.internal.ProxyClassResolver.resolveProxyClass(ProxyClassResolver.java:68)
at weblogic.common.internal.WLObjectInputStream.resolveProxyClass(WLObjectInputStream.java:76)
at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
Truncated. see log file for complete stacktrace
>

Even if you have opened up all relevant ports, the problem persists:

root@soahost1> iptables -S | grep IN_public_allow
-N IN_public_allow
-A IN_public -j IN_public_allow
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 7001 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 8001 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 9001 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 5556 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 7 -m conntrack --ctstate NEW -j ACCEPT

Solution

1. Turn off the local firewall: