Monday, April 10, 2017

Oracle Coherence requires port 7

Problem:

When starting up the SOA server, you find the following repeating every 5 seconds in the logs:
[2017-03-09T18:28:22.148+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@5117501122 3.7.1.13] [userId: <anonymous>] [ecid: 0000LeoJxT4iW1OkPqb000003x73b605zz,1:31280] [APP: soa-infra] 2017-03-09 18:28:22.148/296.247 Oracle Coherence GE 3.7.1.13 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2017-03-09 18:24:34.601, Address=192.168.0.7:8088, MachineId=12345, Location=site:,machine:soahost2,process:4447, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7 

Solution:

1. Open up your firewall and allow port 7 between both your servers.

Port 7 is the default port of the IpMonitor component that is used for detecting hardware failure of cluster members, available in 3.6.0 and later versions of Coherence. I yourt is the default port for the Echo Protocol and, consequently, used by Coherence as the "death detector" port. This means that Coherence uses port 7 to determine if a member of the cluster is reachable (not "dead").


References:

  • Oracle Doc ID 1530288.1
  • Oracle Doc ID 1526745.1
  • Oracle Doc ID 1472388.1


Applicable Version:

  • Oracle SOA Suite 11g (11.1.1.9)


at

12 comments:

  1. AnonymousApril 5, 2018 at 6:47 AM

    I allowed in- and outgoing Connection with TCP Port 7 on my local Windows Firewall, but won't help.

    ReplyDelete
  2. UnknownApril 5, 2018 at 10:18 AM

    Can you paste the errors you're receiving in your logs? And please indicate the name of the log file.

    ReplyDelete
  3. AnonymousApril 6, 2018 at 6:34 AM

    Hi Ahmed,

    thanks for the quick response. Here the log output:

    Every 5 seconds:
    2018-04-06 12:30:24.803/520.291 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2018-04-03 13:25:46.817, Address=130.223.60.249:8088, MachineId=10530, Location=site:,machine:D4790988,process:10080, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7

    Thanks Norman

    ReplyDelete
  4. AnonymousApril 6, 2018 at 6:35 AM

    And after some time this one:

    2018-04-06 12:30:44.193/539.682 Oracle Coherence GE 3.7.1.13 (thread=[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
    MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
    )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=1200000
    RecycleSet=MemberSet(Size=0
    )
    )
    )
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
    at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
    at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
    at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
    at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
    at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
    at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
    at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
    at de.ba.apollo.webservice.ServiceInterceptorImpl.invoke(ServiceInterceptorImpl.java:100)
    at de.ba.apollo.webservice.$Proxy524.getRoleByEKid(Unknown Source)
    at de.ba.apollo.fassaden.ors.OnlineRollenServiceFassade.getRole(OnlineRollenServiceFassade.java:35)
    at de.ba.apollo.term.filter.AllowedRoleFilter.isUserInAllowedRole(AllowedRoleFilter.java:102)
    at de.ba.apollo.term.filter.AllowedRoleFilter.doFilter(AllowedRoleFilter.java:68)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138)
    at java.security.AccessController.doPrivileged(Native Method)
    at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
    at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464)
    at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121)
    at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211)
    at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
    at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
    at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
    at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

    ReplyDelete
  5. AnonymousApril 6, 2018 at 6:36 AM

    2018-04-06 12:30:44.356/539.844 Oracle Coherence GE 3.7.1.13 (thread=[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): TCMP bound to /130.223.60.14:8089 using SystemSocketProvider
    2018-04-06 12:30:44.847/540.335 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMonitor failed to verify the reachability of senior Member(Id=1, Timestamp=2018-04-03 13:25:46.817, Address=130.223.60.249:8088, MachineId=10530, Location=site:,machine:D4790988,process:10080, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7
    2018-04-06 12:30:44.847/540.335 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Failure to join a cluster for 300 seconds; stopping cluster service.
    2018-04-06 12:30:44.852/540.340 Oracle Coherence GE 3.7.1.13 (thread=Cluster, member=n/a): Service Cluster left the cluster

    ReplyDelete
  6. UnknownApril 6, 2018 at 8:01 AM

    Are you starting up Coherence manually? Or is it an Oracle Fusion Middleware product you are starting up? If so, what product?

    ReplyDelete
  7. AnonymousApril 11, 2018 at 3:21 AM

    I don't start the coherence at all.
    It is also a strange behaviour in our team, because we all have the same weblogic running from the same zip and did not have any additionally configuration.
    The half of the team has the problem and the other half not.
    It was working on my maschine and suddenly from one day to another I got this problem.
    We all have a windows 7 machine.

    ReplyDelete
  8. Ahmed AboulnagaApril 11, 2018 at 8:59 PM

    It seems that you have installed Oracle SOA Suite on your laptop (in a single node), perhaps for development purposes.

    1. Did you set any "-Dtangosol.coherence" properties in the startup arguments of your SOA managed server?

    2. If you are running 11g, why do you care about Coherence?

    3. If you are running 12c, are you using the Coherence Adapter? If not, why do you care about Coherence?

    ReplyDelete
  9. AnonymousApril 13, 2018 at 1:38 AM

    I will check it and get back to you. I must also say, I'm not an expert in Weblogic, just have to use it and some collegues set the Weblogic up for local use.

    Thanks for the moment.

    ReplyDelete
  10. AnonymousMay 3, 2018 at 11:25 AM

    So we solved it. There was an caching issue, where the coherence was needed.
    We now deactivated it for local development.

    Thanks a lot.

    ReplyDelete
  11. AnonymousJune 26, 2018 at 10:53 AM

    Can you Explain how to resolve this problem?

    ReplyDelete
  12. megan collinsNovember 24, 2020 at 5:37 AM

    Simply wish to say your article is as astonishing. The clarity in your post is simply great, and I could assume you are an expert on this subject. Same as your blog i found another one Oracle SOA .Actually I was looking for the same information on internet for Oracle SOA Suite and came across your blog. I am impressed by the information that you have on this blog. Thanks a million and please keep up the gratifying work.

    ReplyDelete