com.tangosol.net.RequestPolicyException: No storage-enabled nodes exist for service DistributedCache

Problem

When using the Sample Cache Client Application (query.sh), I am getting an error inserting some data into the cache:

CohQL> insert into "products" key "television" value "ID-5070"

com.tangosol.net.RequestPolicyException: No storage-enabled nodes exist for service DistributedCache

This error occurs because the cache client is configured to start in storage-disabled mode and is unable to access the Cache Server.

Investigation

The Oracle Coherence Cache Server was started up via the cache-server.sh script.

Afterwards, I started up the Sample Cache Client Application:

oracle@soahost1:/u01/app/oracle/Middleware/coherence_3.7/bin> ./query.sh
** Starting storage disabled console **
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1750-linux-x86_64, compiled mode)

Coherence Command Line Tool

CohQL> 

 I then created a cache:

CohQL> create cache "products"
2012-02-20 11:14:04.535/20.168 Oracle Coherence 3.7.1.0 <Info> (thread=Main Thread, member=n/a): Loaded operational configuration from "jar:file:/u01/app/oracle/Middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence.xml"
2012-02-20 11:14:04.588/20.220 Oracle Coherence 3.7.1.0 <Info> (thread=Main Thread, member=n/a): Loaded operational overrides from "jar:file:/u01/app/oracle/Middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-02-20 11:14:04.589/20.221 Oracle Coherence 3.7.1.0 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2012-02-20 11:14:04.598/20.230 Oracle Coherence 3.7.1.0 <D5> (thread=Main Thread, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.7.1.0 Build 27797
 Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

2012-02-20 11:14:05.386/21.018 Oracle Coherence GE 3.7.1.0 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from "jar:file:/u01/app/oracle/Middleware/coherence_3.7/lib/coherence.jar!/coherence-cache-config.xml"
2012-02-20 11:14:09.421/25.053 Oracle Coherence GE 3.7.1.0 <D4> (thread=Main Thread, member=n/a): TCMP bound to /192.168.97.111:8090 using SystemSocketProvider
2012-02-20 11:14:13.986/29.618 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): Created a new cluster "cluster:0xFCDB" with Member(Id=1, Timestamp=2012-02-20 11:14:09.587, Address=192.168.97.111:8090, MachineId=16555, Location=site:,machine:soahost1,process:15138, Role=TangosolCoherenceQueryPlus, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0xC0A8616F000001359B8AA6B340AB1F9A
2012-02-20 11:14:14.040/29.672 Oracle Coherence GE 3.7.1.0 <Info> (thread=Main Thread, member=n/a): Started cluster Name=cluster:0xFCDB

Group{Address=224.3.7.0, Port=37000, TTL=4}

MasterMemberSet(
  ThisMember=Member(Id=1, Timestamp=2012-02-20 11:14:09.587, Address=192.168.97.111:8090, MachineId=16555, Location=site:,machine:soahost1,process:15138, Role=TangosolCoherenceQueryPlus)
  OldestMember=Member(Id=1, Timestamp=2012-02-20 11:14:09.587, Address=192.168.97.111:8090, MachineId=16555, Location=site:,machine:soahost1,process:15138, Role=TangosolCoherenceQueryPlus)
  ActualMemberSet=MemberSet(Size=1
    Member(Id=1, Timestamp=2012-02-20 11:14:09.587, Address=192.168.97.111:8090, MachineId=16555, Location=site:,machine:soahost1,process:15138, Role=TangosolCoherenceQueryPlus)
    )
  MemberId|ServiceVersion|ServiceJoined|MemberState
    1|3.7.1|2012-02-20 11:14:13.99|JOINED
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0
    )
  )

TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}

2012-02-20 11:14:14.471/30.103 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2012-02-20 11:14:15.584/31.216 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1

But if you notice from the output above, we are senior service member 1 here. That's odd. That's an indication that it couldn't find the Cache Server.

Naturally, when I try to insert data, I will get the following exception:

CohQL> insert into "products" key "television" value "ID-5070"

com.tangosol.net.RequestPolicyException: No storage-enabled nodes exist for service DistributedCache

This is strange because when using coherence.sh, everything is working fine.

Solution

As expected, the Sample Cache Client Application (query.sh) is not able to connect to the Cache Server (cache-server.sh).

By observing the value of JAVA_OPTS in each of the scripts...

cache-server.sh
JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY $JMXPROPERTIES -Dtangosol.coherence.cluster=coh_cluster -Dtangosol.coherence.clusterport=9876"

coherence.sh
JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY $JMXPROPERTIES -Dtangosol.coherence.cluster=coh_cluster -Dtangosol.coherence.clusterport=9876"

query.sh
JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.distributed.localstorage=$STORAGE_ENABLED"

Both cache-server.sh and coherence.sh were using a cluster name of "coh_cluster" and a cluster port of "9876" while query.sh was unaware of this.

1. Simply add this to query.sh to mimic the Cache Server setting:

JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.distributed.localstorage=$STORAGE_ENABLED -Dtangosol.coherence.cluster=coh_cluster -Dtangosol.coherence.clusterport=9876"

Applicable Versions:

• Oracle Coherence 3.7.1