Monday, April 10, 2017

Oracle Coherence requires port 7

Problem:

When starting up the SOA server, you find the following repeating every 5 seconds in the logs:
[2017-03-09T18:28:22.148+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@5117501122 3.7.1.13] [userId: <anonymous>] [ecid: 0000LeoJxT4iW1OkPqb000003x73b605zz,1:31280] [APP: soa-infra] 2017-03-09 18:28:22.148/296.247 Oracle Coherence GE 3.7.1.13 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2017-03-09 18:24:34.601, Address=192.168.0.7:8088, MachineId=12345, Location=site:,machine:soahost2,process:4447, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7 

Solution:

1. Open up your firewall and allow port 7 between both your servers.

Port 7 is the default port of the IpMonitor component that is used for detecting hardware failure of cluster members, available in 3.6.0 and later versions of Coherence. I yourt is the default port for the Echo Protocol and, consequently, used by Coherence as the "death detector" port. This means that Coherence uses port 7 to determine if a member of the cluster is reachable (not "dead").


References:

  • Oracle Doc ID 1530288.1
  • Oracle Doc ID 1526745.1
  • Oracle Doc ID 1472388.1


Applicable Version:

  • Oracle SOA Suite 11g (11.1.1.9)


11 comments:

Anonymous said...

I allowed in- and outgoing Connection with TCP Port 7 on my local Windows Firewall, but won't help.

Ahmed Aboulnaga said...

Can you paste the errors you're receiving in your logs? And please indicate the name of the log file.

Anonymous said...

Hi Ahmed,

thanks for the quick response. Here the log output:

Every 5 seconds:
2018-04-06 12:30:24.803/520.291 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2018-04-03 13:25:46.817, Address=130.223.60.249:8088, MachineId=10530, Location=site:,machine:D4790988,process:10080, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7

Thanks Norman

Anonymous said...

And after some time this one:

2018-04-06 12:30:44.193/539.682 Oracle Coherence GE 3.7.1.13 (thread=[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
)
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at de.ba.apollo.webservice.ServiceInterceptorImpl.invoke(ServiceInterceptorImpl.java:100)
at de.ba.apollo.webservice.$Proxy524.getRoleByEKid(Unknown Source)
at de.ba.apollo.fassaden.ors.OnlineRollenServiceFassade.getRole(OnlineRollenServiceFassade.java:35)
at de.ba.apollo.term.filter.AllowedRoleFilter.isUserInAllowedRole(AllowedRoleFilter.java:102)
at de.ba.apollo.term.filter.AllowedRoleFilter.doFilter(AllowedRoleFilter.java:68)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Anonymous said...

2018-04-06 12:30:44.356/539.844 Oracle Coherence GE 3.7.1.13 (thread=[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): TCMP bound to /130.223.60.14:8089 using SystemSocketProvider
2018-04-06 12:30:44.847/540.335 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2018-04-03 13:25:46.817, Address=130.223.60.249:8088, MachineId=10530, Location=site:,machine:D4790988,process:10080, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7
2018-04-06 12:30:44.847/540.335 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Failure to join a cluster for 300 seconds; stopping cluster service.
2018-04-06 12:30:44.852/540.340 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Service Cluster left the cluster

Ahmed Aboulnaga said...

Are you starting up Coherence manually? Or is it an Oracle Fusion Middleware product you are starting up? If so, what product?

Anonymous said...

I don't start the coherence at all.
It is also a strange behaviour in our team, because we all have the same weblogic running from the same zip and did not have any additionally configuration.
The half of the team has the problem and the other half not.
It was working on my maschine and suddenly from one day to another I got this problem.
We all have a windows 7 machine.

Ahmed Aboulnaga said...

It seems that you have installed Oracle SOA Suite on your laptop (in a single node), perhaps for development purposes.

1. Did you set any "-Dtangosol.coherence" properties in the startup arguments of your SOA managed server?

2. If you are running 11g, why do you care about Coherence?

3. If you are running 12c, are you using the Coherence Adapter? If not, why do you care about Coherence?

Anonymous said...

I will check it and get back to you. I must also say, I'm not an expert in Weblogic, just have to use it and some collegues set the Weblogic up for local use.

Thanks for the moment.

Anonymous said...

So we solved it. There was an caching issue, where the coherence was needed.
We now deactivated it for local development.

Thanks a lot.

Anonymous said...

Can you Explain how to resolve this problem?