Thursday, June 20, 2019

Cannot start up WebLogic and NodeManager after Oracle Cloud data center outage

Problem

When starting Oracle WebLogic Server 11g AdminServer on an Oracle Cloud Infrastructure virtual machine, the WebLogic process was dying at startup. No explanation whatsoever as shown here:
INFO: Notification sent for Mapping config object reloaded 
<Jul 16, 2018, 8:20:01,49 PM EDT> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://adminhost.raastech.com:7002/jndi/weblogic.management.mbeanservers.editsession.DOMAIN.ServiceBus.> 
<Jul 16, 2018, 8:20:01,73 PM EDT> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://adminhost.raastech.com:7002/jndi/weblogic.management.mbeanservers.edit.> 
<Jul 16, 2018, 8:20:01,147 PM EDT> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://adminhost.raastech.com:7002/jndi/weblogic.management.mbeanservers.domainruntime.> 
Jul 16, 2018 8:20:01 PM oracle.ods.virtualization.engine.util.VDELogger info INFO: Notification sent for Mapping config object reloaded /u01/app/oracle/middleware/products/fmw1221/user_projects/domains/soa_domain/bin/startWebLogic.sh: line 203: 24671 Killed ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS} 
Stopping Derby server... 
Derby server stopped.
When starting up NodeManager, observed the following errors:
<Jul 16, 2018 8:21:11 PM EDT> <SEVERE> <Fatal error in NodeManager server>
java.io.IOException: The server is unable to create a server socket for listening. The address soahost1.raastech.com might be incorrect or another process is using port 5556: java.net.BindException: Cannot assign requested address
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:351)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:529)
        at weblogic.NodeManager.main(NodeManager.java:31)
Caused by: java.net.BindException: Cannot assign requested address
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:181)
        at sun.security.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:140)
        at sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:95)
        at weblogic.nodemanager.server.SSLListener.privilegedInit(SSLListener.java:134)
        at weblogic.nodemanager.server.SSLListener.access$000(SSLListener.java:33)
        at weblogic.nodemanager.server.SSLListener$1.run(SSLListener.java:97)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLListener.init(SSLListener.java:94)
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:349)
        ... 2 more
Failed attempts at resolution:

  • Started WebLogic manually. No help.
  • Increased debugging. No additional information.
  • Reset the 'weblogic' password. No help.
  • Cleared the ~/tmp ~/cache ~/data folders of the AdminServer. No help.
  • Restored from backup. No help.
  • This behavior appeared to be attributed to memory issues, so shut down every single service on the box. No help.

Note that there was an outage at the Oracle Cloud data center, which killed our VM. All problems started happening after the VM was started up.

Solution

1. Memory hardware issue. Rebuild the VM.

For example:
a. Stop the VM.
b. Create a new VM.
c. Mount the storage volumes on the new VM.
d. Rename the hostname of the new VM.
e. Everything will start up smoothly with no problems.


Applicable Versions
  • Oracle WebLogic Server (10.3.6)
  • Oracle Cloud Infrastructure (2019)

 

Monday, June 3, 2019

DNS name resolution issue in OCI Classic

Problem

In your Oracle Cloud Infrastructure Classic virtual machine, you may experience DNS resolution issues to *.oracle.com domains, similar to what is described at https://community.oracle.com/thread/3944301.


Solution

1. Edit /etc/resolve.conf.

2. Add the following entry:
nameserver 8.8.8.8

References



Applicable Versions
  • Oracle Cloud Infrastructure Classic (2019)



Thursday, May 2, 2019

WSM-02120 when attaching or deploying OWSM policy

Problem

When attaching an Oracle Web Services Manager (OWSM) policy or deploying a new JAR with an OWSM policy, you may encounter the following exception:
[OSB-387178]An error occurred while computing the web service policy for service SBRestTest/PipelineProxyService: com.bea.wli.sb.security.wss.PolicyManagementException: oracle.wsm.policymanager.PolicyManagerException: WSM-02120 : Unable to connect to the policy access service. [Possible Cause : No available router to destination.] [No available router to destination.]

Solution

1. Start up the WSM managed server.


Applicable Versions
  • Oracle Web Services Manager 11g (11.1.1.x)

 

Wednesday, March 20, 2019

Getting java.lang.ExceptionInInitializerError during OSB deployment

Problem

During the deployment of the OSB project from the command line, we get the following cryptic error:
Diagnostic XML Bean debug log file created: /tmp/xmlbeandebug5401068355720061527.log
* Unexpected error: java.lang.ExceptionInInitializerError
* Session discarded
* Unexpected error:  java.lang.ExceptionInInitializerError
No stack trace available.

Problem invoking WLST - Traceback (innermost last):
  File "/home/oracle/deploy/osb_import.py", line 277, in ?
  File "/home/oracle/deploy/osb_import.py", line 134, in importToALSBDomain
        at sun.misc.Unsafe.ensureClassInitialized(Native Method)
        at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
        at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:140)
        at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1057)
        at java.lang.reflect.Field.getFieldAccessor(Field.java:1038)
        .
        .
        .
java.lang.ExceptionInInitializerError: java.lang.ExceptionInInitializerError
Opening up the .log file in /tmp doesn't help much and only yields more cryptic information:
XML-BEANS compiled schema: Could not locate compiled schema resource schemaorg_apache_xmlbeans/system/resources_schema_xbtypes/index.xsb (schemaorg_apache_xmlbeans.system.resources_schema_xbtypes.index) - code 0
org.apache.xmlbeans.SchemaTypeLoaderException: XML-BEANS compiled schema: Could not locate compiled schema resource schemaorg_apache_xmlbeans/system/resources_schema_xbtypes/index.xsb (schemaorg_apache_xmlbeans.system.resources_schema_xbtypes.index) - code 0
        at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl$XsbReader.<init>(SchemaTypeSystemImpl.java:1519)
        at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl.initFromHeader(SchemaTypeSystemImpl.java:273)
        at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl.<init>(SchemaTypeSystemImpl.java:185)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

Solution

1. Try importing the JAR through the OSB Console, and you will get a more accurate error as to why the deployment failed.

Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

Saturday, March 9, 2019

BEA-473003 Destination unreachable on t3s; nested exception is: java.net.NoRouteToHostException

Problem

Seeing these errors in osb_server1.out log file (note the T3S protocol in the log):
<Mar 29, 2018 4:14:00 PM GMT> <Error> <ALSB Statistics Manager> <BEA-473003> <Aggregation Server Not Available. Failed to get remote aggregator
javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://192.168.0.30:8011: Destination unreachable; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); No available router to destination]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:368)
        at weblogic.jndi.Environment.getContext(Environment.java:315)
        at weblogic.jndi.Environment.getContext(Environment.java:285)
        Truncated. see log file for complete stacktrace
Caused By: java.net.ConnectException: t3s://192.168.0.30:8015: Destination unreachable; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); No available router to destination
        at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
        at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
        at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:353)
        at weblogic.jndi.Environment.getContext(Environment.java:315)
        Truncated. see log file for complete stacktrace
Caused By: java.rmi.ConnectException: Destination unreachable; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); No available router to destination
        at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)
        at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321)
        at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:262)
        at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:199)
        at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
        Truncated. see log file for complete stacktrace
>
Solution

1. Log in to the WebLogic Admin Console

2. Navigate to [managed server] > SSL > Advanced

3. Set "Hostname Verification" to "None"

4. Repeat for all managed servers

5. Restart the managed servers


Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Monday, February 18, 2019

BEA-423223 There was an error while running a lifecycle stage

Problem

When deploying an OSB project, you may get the following undescriptive error on the OSB Console:
An unexpected error occurred.
The following error would appear in the AdminServer.out file:
####<Dec 3, 2018 5:03:21 PM GMT> <Error> <netuix> <osbhost1> <AdminServer> <[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <685ea1de428af539:-5fcff8e9:82f2515a61f:-8000-0000000000003938> <1488560601009> <BEA-423137> <There was an error loading the requested URI null.> 
####<Dec 3, 2018 5:03:21 PM GMT> <Error> <netuix> <osbhost1> <AdminServer> <[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <de428af5685ea139:-5fcff8e9:15a61f82f25:-8000-0000000000003938> <1488560601009> <BEA-423223> <There was an error while running a lifecycle stage :: Lifecycle: UIControl.render :: for the control :: null ::.
com.bea.netuix.nf.UIControlException: com.bea.portlet.adapter.scopedcontent.ActionLookupFailedException: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.readFully(Ljava/io/InputStream;[B)V
        at com.bea.netuix.servlets.controls.content.NetuiContent.checkPreRenderExceptions(NetuiContent.java:404)
        at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:343)
        at com.bea.netuix.nf.ControlLifecycle$7.visit(ControlLifecycle.java:485)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:518)
Solution

1. The OSB JAR is missing the ExportInfo file. The developer must fix this..


Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Thursday, January 17, 2019

BEA-000337 and STUCK thread in OSB 11g

Problem

Seeing the AdminServer in warning state and a stuck thread. AdminServer.log shows this:
####<Dec 15, 2018 4:54:00 PM GMT> <Error> <WebLogicServer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004fb0> <1489596840017> <BEA-000337> <[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,433" seconds working on the request "Workmanager: weblogic.kernel.Default, Version: 0, Scheduled=false, Started=true, Started time: 1434264 ms
", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
null>
Solution

1. Look through the AdminServer.log log file. Could be related to an OutOfMemory exception that occurred during deployment.
####<Dec 15, 2018 4:32:00 PM GMT> <Warning> <Deployer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004f91> <1489595520836> <BEA-149078> <Stack trace for message 149004
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.notifyPrepareFailure(ServerDeploymentReceiver.java:355)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver._prepare(ServerDeploymentReceiver.java:434)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.access$700(ServerDeploymentReceiver.java:70)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver$1.run(ServerDeploymentReceiver.java:246)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.prepare(ServerDeploymentReceiver.java:213)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.lang.String.substring(String.java:1877)

Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Saturday, January 5, 2019

HTTP 500 when logging in to Siebel with SSO/X509

Problem

When accessing the Siebel application at: https://siebelhost1.raastech.com/epublicsector_enu/, you get an HTTP 500 on the browser.

All services are up.

Nothing is in any of these logs:
  • WebGate --> C:\NetPoint\Webgate\access\oblix\logs\oblog.log
  • Siebel --> C:\Siebel\15.0.0.0.0\eappweb\log\ss170601_2516.log
  • IIS --> C:\inetpub\logs\LogFiles\W3SVC1\u_ex170606.log
Solution

1. There was likely an IAM outage. You will need to restart the SIE01 and SIE02 webtiers.


Applicable Versions
  • Oracle Access Manager 11g

 

Tuesday, January 1, 2019

BEA-000403 MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds

Problem

Error received during OSB deployment:
<Jan 2, 2018 7:13:01 PM GMT> <Error> <Socket> <BEA-000403> <IOException occurred on socket: Socket[addr=osbhost1.raastech.com/192.168.0.30,port=7001,localport=57684]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
        at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
        at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:369)
        at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:984)
        at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
        at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:887)
        at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:343)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
>
 
Solution

1. Navigate to Servers > AdminServer > Protocols and "Enable Tunneling"

2. Make this change to the envcode.properties file (or equivalent) when deploying (basically, use HTTP instead of T3):
OLD: osbAdminUrl=t3://192.168.0.30:7001

NEW: osbAdminUrl=http://192.168.0.30:7001

Applicable Versions
  • Oracle WebLogic Server 11g (11.1.1.9.0)
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Wednesday, December 5, 2018

Getting "Unable to create SSL Socket Factory" when starting up SOA 11.1.1.9 on SSL

Problem

All of a sudden, when we shut down and restart "bam_server1", the SSL port no longer works. This is the error that shows up when we start it up:
<May 9, 2018 4:49:06 AM GMT> <Critical> <Security> <BEA-090174> <The trust keystore configuration specified on the command line or in boot.properties does not match the trust keystore configuration specified in config.xml.>  
Solution

1. Check "bam_server1.out" and note the JVM start parameters. You will notice that they conflict with what you intend.

Someone may have made custom SSL settings in this files that conflicted with the configuration we made on the WebLogic Console. (In our case, is was 3 "ssl" arguments in EXTRA_JAVA_PROPERTIES.) Check these files:
  • $MSERVER_HOME/bin/setDomainEnv.sh
  • $ASERVER_HOME/bin/setDomainEnv.sh
  • $MSERVER_HOME/bin/startManagedWebLogic.sh
  • $ASERVER_HOME/bin/startManagedWebLogic.sh

Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Saturday, November 24, 2018

ORABPEL-9732 'Persistency service internal error: Installation failed' during startup

Problem

During the startup of the SOA managed servers, you see the following ORABPEL-9732 error and multiple "Installation failed" errors:
<Apr 1, 2018 3:30:00 PM GMT> <Notice> <Stdout> <BEA-000000> <<Apr 1, 2018 3:30:00 PM GMT> <Error> <oracle.soa.services.workflow.common> <BEA-000000> <<.> Persistency service internal error: Persistency service internal error: Installation failed.
Persistency service experienced unexpected internal error.
Check the underlying exception and correct the error. If the error persists, contact Oracle Support Services.
Persistency service experienced unexpected internal error.
Check the underlying exception and correct the error. If the error persists, contact Oracle Support Services.
      
ORABPEL-9732
Solution

1. Restore the database.

There is a problem with the database. It could be empty tables, missing sequences, or invalid objects. Don't waste your time troubleshooting and restore the database from a working copy.


Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Friday, November 2, 2018

Getting "Invalid server bam_server1 SSL configuration" when starting up BAM 11.1.1.9 on SSL

Problem

All of a sudden, when we shut down and restart "bam_server1", the SSL port no longer works. This is the error that shows up when we start it up:
<Aug 9, 2018 4:49:06 AM GMT> <Critical> <Security> <BEA-090174> <The trust keystore configuration specified on the command line or in boot.properties does not match the trust keystore configuration specified in config.xml.>  
<Aug 9, 2018 4:49:06 AM GMT> <Alert> <Security> <BEA-090717> <Invalid server bam_server1 SSL configuration>  
<Aug 9, 2018 4:49:06 AM GMT> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, weblogic.management.configuration.ConfigurationException: Invalid server bam_server1 SSL configuration>  
<Aug 9, 2018 4:49:06 AM GMT> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Invalid server bam_server1 SSL configuration.>
Solution

1. Check bam_server1.out and note the JVM start parameters. You will notice that they conflict with what you intend.

2. Someone may have made custom SSL settings in this files that conflicted with the configuration we made on the WebLogic Console, so check these files too:
  • setDomainEnv.sh
  • startManagedServer.sh

Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Sunday, September 30, 2018

BEA-000297 Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate

Problem

You try to configure SSL on the managed server, but it is not working and you get the following error in the log:
<Apr 16, 2018 5:57:38 PM GMT> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias soahost1.raastech.com from the JKS keystore file /u01/app/oracle/middleware/keystores/raastech.jks.>
<Apr 16, 2018 5:57:38 PM GMT> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>
Solution

1. Navigate to [managed server] > Configuration > SSL > Advanced.

2. Check "Use JSSE SSL".

3. Save and Activate changes.

Applicable Versions
  • Oracle WebLogic Server 11g (11.1.1.9.0)

 

Tuesday, September 25, 2018

Cannot resolve reference to bean 'DOStoreFactory' during SOA startup

Problem

When starting up "soa_server1" it comes up fine. But when we start "soa_server2", we get these errors in the soa_server2.out log file:
<Mar 2, 2018 1:39:56 PM GMT> <Warning> <HTTP> <BEA-101162> <User defined listener com.oracle.weblogic.wsee.wrapper.org.springframework.web.context.ContextLoaderListener failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SensorManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricMesh' while setting bean property 'fabricMesh'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMesh': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMeshTarget' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'WSServiceBC' while setting bean property 'bindingComponents' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'WSServiceBC' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'InstanceManager' while setting bean property 'instanceManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'InstanceManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'DOStoreFactory' while setting bean property 'DOStoreFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DOStoreFactory' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricConfigManager' while setting bean property 'fabricConfigManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricConfigManager' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'EventBusConfigurable' while setting bean property 'configurables' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'EventBus' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'ClusterInterface' while setting bean property 'clusterInterface'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ClusterInterface' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
 MemberSet=MasterMemberSet(
 ThisMember=Member(Id=2, Timestamp=2018-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
 OldestMember=Member(Id=1, Timestamp=2018-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
 ActualMemberSet=MemberSet(Size=2
 Member(Id=1, Timestamp=2018-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
 Member(Id=2, Timestamp=2018-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
 )
 MemberId|ServiceVersion|ServiceJoined|MemberState
 1|3.7.1|2018-05-02 13:34:29.559|JOINING,
 2|3.7.1|2018-05-02 13:34:56.523|JOINED
 RecycleMillis=1200000
 RecycleSet=MemberSet(Size=0
 )
 )
 ).
You may also see this error:
<Mar 2, 2018 1:41:26 PM GMT> <Error> <Cluster> <BEA-000140> <Failed to deserialize statedump from server 6011237482029920459S:192.168.0.21:[8001,8001,-1,-1,-1,-1,-1]:soahost1:8001,soahost2:8001:soadomain:soa_server2 with java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote.
 java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote
 at weblogic.application.internal.AppClassLoaderManagerImpl.loadApplicationClass(AppClassLoaderManagerImpl.java:164)
 at weblogic.common.internal.ProxyClassResolver.resolveProxyClass(ProxyClassResolver.java:68)
 at weblogic.common.internal.WLObjectInputStream.resolveProxyClass(WLObjectInputStream.java:76)
 at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
 Truncated. see log file for complete stacktrace
 >
Even if you have opened up all relevant ports, the problem persists:
root@soahost1> iptables -S | grep IN_public_allow
 -N IN_public_allow
 -A IN_public -j IN_public_allow
 -A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 7001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 8001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 9001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 5556 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 7 -m conntrack --ctstate NEW -j ACCEPT
Solution

1. Turn off the local firewall:
service firewalld stop
You may need to spend more time to open up UDP and multicast on every NIC to re-enable the local firewall.

References
Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Saturday, June 30, 2018

BEA-149205 Failed to initialize the application '_ALSB'

Problem

Getting the following error when starting up the OSB managed server. Multiple applications are failing, and this is just a snippet of one of the errors:
<Feb 28, 2018 12:38:45 PM GMT> <Error> <Deployer> <BEA-149205> <Failed to initialize the application '_ALSB_1491592089579' due to error weblogic.management.DeploymentException: Exception occured while downloading files.
weblogic.management.DeploymentException: Exception occured while downloading files
        at weblogic.deploy.internal.targetserver.datamanagement.AppDataUpdate.doDownload(AppDataUpdate.java:43)
        at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
        at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:97)
        at weblogic.deploy.internal.targetserver.BasicDeployment.prepareDataUpdate(BasicDeployment.java:704)
        at weblogic.deploy.internal.targetserver.BasicDeployment.stageFilesForStatic(BasicDeployment.java:747)
        Truncated. see log file for complete stacktrace
Caused By: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from admin server for deployment request "0". Underlying error is: "null"
        at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:86)
        at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:165)
        at weblogic.deploy.internal.targetserver.datamanagement.AppDataUpdate.doDownload(AppDataUpdate.java:39)
        at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
        at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:97)
        Truncated. see log file for complete stacktrace
>
 
Solution

1. You must undeploy these applications

2. Start up the AdminServer

3. Connect through WLST and run the following. Make sure to undeploy the applications that are showing up in the logs, so do not use the same exact values here.
cd /u01/app/oracle/middleware/wlserver_10.3/common/bin/ 
./wlst.sh 
connect( 'weblogic', 'welcome1', 't3://localhost:7001') 
stopApplication('_ALSB_1492546342692') 
undeploy('_ALSB_1492546342692')  
stopApplication('_ALSB_1492545743924')
undeploy('_ALSB_1492545743924')
stopApplication('_ALSB_1491592307429')
undeploy('_ALSB_1491592307429')
stopApplication('_ALSB_1491592306429')
undeploy('_ALSB_1491592306429')
stopApplication('_ALSB_1491592089579')
undeploy('_ALSB_1491592089579')
disconnect()
4. Login to WebLogic Admin Console and activate the session

References
Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Tuesday, June 19, 2018

BEA-240003 java.net.NoRouteToHostException: No route to host (Host unreachable)

Problem

When trying to configure SSL and activating your changes on the WebLogic Admin Console, you get the following error on both the console and in the logs:
<Mar 18, 2018 4:17:04 PM GMT> <Error> <Console> <BEA-240003> <Console encountered the following error java.rmi.RemoteException: [Deployer:149150]An IOException occurred while reading input.; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable)

        .
        .
        .
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:344)
Solution

1. This is because the trust.jks you are using is missing the correct chain.

For example, if your identity cert is issued by a CA, you must import that CA intermediate and root certs into trust.jks:
keytool -import -alias PRODCA4 -file PRODCA4.crt -keystore /u01/app/oracle/middleware/keystores/trust.jks

keytool -import -alias PRODTreasury -file PRODTreasury.crt -keystore /u01/app/oracle/middleware/keystores/trust.jks

Applicable Versions
  • Oracle WebLogic server 11g (11.1.1.9.0)

 

Tuesday, December 26, 2017

Cloning OID 11g

Oracle does not support the changing of hostname or IP address for your Oracle Internet Directory 11g installation, rendering cloning impossible. These instructions, however, guide you through that process.


1. Set the environment:
export ORACLE_HOME=/u01/app/oracle/middleware/Oracle_OID
export ORACLE_INSTANCE=/u01/app/oracle/middleware/asinst_1
export TNS_ADMIN=$ORACLE_INSTANCE/config 
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/ldap/bin:$ORACLE_INSTANCE/bin:$PATH
2. Clear the temp folders on node 1 (for AdminServer, wls_ods1):
cd ${ASERVER_HOME}/servers/AdminServer
rm -rf cache/* logs/* tmp/*
cd ${MSERVER_HOME}/servers/wls_ods1
rm -rf cache/* logs/* tmp/* data/*
cd ${MSERVER_HOME}/servers/wls_ods2 
rm -rf cache/* logs/* tmp/* data/*
3. Replace values in red:
cd /u01/app/oracle/middleware
find . -type f \( ! -name "oidpwdr*" ! -name "*.DAT" ! -name "*.cer" ! -name "*dump1" ! -name "*.swp" ! -name "*.jks" \) -exec sed -i "s/OLDHOST/NEWHOST/g" {} \;
find . -type f \( ! -name "oidpwdr*" ! -name "*.DAT" ! -name "*.cer" ! -name "*dump1" ! -name "*.swp" ! -name "*.jks" \) -exec sed -i "s/OLDHOST/NEWHOST/g" {} \; 
find . -type f \( ! -name "oidpwdr*" ! -name "*.DAT" ! -name "*.cer" ! -name "*dump1" ! -name "*.swp" ! -name "*.jks" \) -exec sed -i "s/.olddomain.com/.newdomain.com/g" {} \;
4. Change the strings to the new database string:
FILE: /u01/app/oracle/middleware/asinst_1/config/tnsnames.ora
OLD: OIDDB=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=prod-cluster)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=oiddb.raastech.com)))
FILE: /u01/app/oracle/middleware/user_projects/domains/oiddomain/config/jdbc/schedulerDS-*-jdbc.xml
OLD: <url>jdbc:oracle:thin:@//prod-cluster:1521/oiddb.raastech.com</url>
OLD: <value>1521</value>
OLD: <value>prod-cluster</value>
5. Run the change-ip-host script:
nohup ${ASERVER_HOME}/bin/startWebLogic.sh >> ${ASERVER_HOME}/servers/${ADMINSERVER}/logs/${ADMINSERVER}.out & 
cd $ORACLE_HOME 
$ORACLE_HOME/chgip/scripts/chgiphost.sh -oldhost OLDHOST -newhost NEWHOST -instanceHome /u01/app/oracle/middleware/asinst_1 
$ORACLE_INSTANCE/bin/opmnctl unregisterinstance -instanceName asinst_1 -adminHost NEWHOST.raastech.com -adminPort 7001 
$ORACLE_INSTANCE/bin/opmnctl redeploy -adminHost NEWHOST.raastech.com -adminPort 7001 
ll /u01/app/oracle/middleware/user_projects/domains/oiddomain/servers/AdminServer/upload/NonJ2EEManagement/11.1.1/app/NonJ2EEManagement.ear 
$ORACLE_INSTANCE/bin/opmnctl redeploy -adminHost NEWHOST.raastech.com -adminPort 7001

$ORACLE_INSTANCE/bin/opmnctl registerinstance -adminHost NEWHOST.raastech.com -adminPort 7001
If the 'redeploy' or 'registerinstance' actions fail, do not worry. Just confirm that "NonJ2EEManagement" is a deployed application and exists in /u01/app/oracle/middleware/user_projects/domains/oiddomain/servers/AdminServer/upload/NonJ2EEManagement/11.1.1/app/NonJ2EEManagement.ear.
 6. Remove the states folder to avoid the "Failed to open locale state file" error:
rm -rf /u01/app/oracle/middleware/asinst_1/config/OPMN/states
7. Reset the ODS password in the wallet (make sure this password matches that of the ODS user in the database):
cd $ORACLE_INSTANCE/OID/admin

mv oidpwdlldap1 oidpwdlldap1.old

$ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB create_wallet=true
8. Startup the rest of the environment:
nohup ${MSERVER_HOME}/bin/startManagedWebLogic.sh wls_ods1 >> ${MSERVER_HOME}/servers/wls_ods1/logs/wls_ods1.out & 
${MW_HOME}/asinst_1/bin/opmnctl startall
9. Verify the installation:
$ORACLE_HOME/bin/ldapbind -h oidhost1.raastech.com -p 3060 -D cn=orcladmin -w welcome1
$ORACLE_HOME/bin/ldapbind -h oidhost1.raastech.com -p 3131 -D cn=orcladmin -w welcome1 -U 1 
$ORACLE_HOME/bin/ldapsearch -h oidhost1.raastech.com -p 3131 -U 1 -D cn=orcladmin -s sub "(objectclass=*)" -w welcome1

Applicable Versions
  • Oracle Internet Directory (OID) 11g (11.1.1.9.0)

 

Wednesday, December 13, 2017

Unable to Connect to Database: Incorrect location for tnsnames.ora in OID 11g

Problem

If you are trying to execute the oidpasswd command, for example:
$ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB create_wallet=true 
$ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB change_oiddb_pwd=true
You receive the following error on the prompt:
Unable to Connect to Database: Incorrect location for tnsnames.ora (derived from ORACLE_INSTANCE) or Incorrect TNS Connect string or Invalid Password

Solution

1. This is because you must set the TNS_ADMIN environment variable:
export ORACLE_HOME=/u01/app/oracle/middleware/Oracle_OID 
export ORACLE_INSTANCE=/u01/app/oracle/middleware/asinst_1 
export TNS_ADMIN=$ORACLE_INSTANCE/config 
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/ldap/bin:$ORACLE_INSTANCE/bin:$PATH
2. If this does not work, try the following:
cp $TNS_ADMIN/tnsnames.ora $ORACLE_HOME/network/admin
3. If this does not work, the problem is with the "ods" account on the database.
a. Try to connect to the "ods" user using Oracle SQL Developer.

b. Try to connect to the "ods" user using SQL*Plus on the OAM server:  
sqlplus ods/welcome1@oiddb
c. Then try resetting the password again or create a new wallet

 
Applicable Versions
  • Oracle Internet Directory (OID) 11g

 

Saturday, December 2, 2017

Test Coherence communication between nodes using 'multicast-test.sh'

This is a successful execution of the multicast-test.sh script. Any other response may indicate communication issues between the nodes.
oracle@soahost1:/u01/app/oracle/middleware/coherence_3.7/bin> ./multicast-test.sh -local soahost1 
java version "1.7.0_121"
Java(TM) SE Runtime Environment (build 1.7.0_121-b32)
Java HotSpot(TM) 64-Bit Server VM (build 24.121-b32, mixed mode)
 
2017-05-03 19:21:53.805/0.380 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence.xml"
2017-05-03 19:21:53.878/0.453 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2017-05-03 19:21:53.878/0.453 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2017-05-03 19:21:53.881/0.456 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
 
Oracle Coherence Version 3.7.1.1 Build 28901
 Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
 
Starting test on ip=soahost1/192.168.0.20, group=/237.0.0.1:9000, ttl=4
Configuring multicast socket...
Starting listener...
Wed May 03 19:21:53 GMT 2017: Sent packet 1 containing 1468 bytes.
Wed May 03 19:21:54 GMT 2017: Received test packet 1 from self (sent 7ms ago).
Wed May 03 19:21:54 GMT 2017: Received test packet 3 from ip=soahost2/192.168.0.21, group=/237.0.0.1:9000, ttl=4 containing 1468 bytes.
Wed May 03 19:21:56 GMT 2017: Sent packet 2 containing 1468 bytes.
Wed May 03 19:21:56 GMT 2017: Received test packet 2 from self (sent 1ms ago).
Wed May 03 19:21:56 GMT 2017: Received test packet 4 from ip=soahost2/192.168.0.21, group=/237.0.0.1:9000, ttl=4 containing 1468 bytes.
Wed May 03 19:21:58 GMT 2017: Sent packet 3 containing 1468 bytes.
Wed May 03 19:21:58 GMT 2017: Received test packet 3 from self (sent 3ms ago).
Wed May 03 19:21:58 GMT 2017: Received test packet 5 from ip=soahost2/192.168.0.21, group=/237.0.0.1:9000, ttl=4 containing 1468 bytes.
Wed May 03 19:22:00 GMT 2017: Sent packet 4 containing 1468 bytes.
Wed May 03 19:22:00 GMT 2017: Received test packet 4 from self (sent 1ms ago).
Wed May 03 19:22:02 GMT 2017: Sent packet 5 containing 1468 bytes.
Wed May 03 19:22:02 GMT 2017: Received test packet 5 from self (sent 1ms ago).
 
Applicable Versions
  • Oracle Coherence 11g (11.1.1.9.0)

 

Friday, December 1, 2017

Test Coherence communication between nodes using 'datagram-test.sh'

This is a successful execution of the datagram-test.sh script. Any other response may indicate communication issues between the nodes.
oracle@soahost1:/u01/app/oracle/middleware/coherence_3.7/bin> ./datagram-test.sh -polite soahost2 
java version "1.7.0_121"
Java(TM) SE Runtime Environment (build 1.7.0_121-b32)
Java HotSpot(TM) 64-Bit Server VM (build 24.121-b32, mixed mode)
2017-05-03 19:19:37.342/0.319 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence.xml"
2017-05-03 19:19:37.412/0.389 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2017-05-03 19:19:37.412/0.389 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2017-05-03 19:19:37.415/0.392 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
 
Oracle Coherence Version 3.7.1.1 Build 28901
 Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
 
creating datagram socket using provider: SystemSocketProvider
starting listener: at /192.168.0.20:9999
packet size: 1468 bytes
buffer size: 1428 packets
  report on: 100000 packets, 139 MBs
    process: 20 bytes/packet
        log: null
     log on: 139 MBs
 
starting publisher: at /192.168.0.20:10000 sending to soahost2/192.168.0.21:9999
packet size: 1468 bytes
buffer size: 32 packets
  report on: 100000 packets, 139 MBs
    process: 20 bytes/packet
      peers: 1
       rate: no limit
 
no packet burst limit
waiting for listener to be contacted before publishing
 
Recieving data from 1 publisher(s).
ioioioioioioiioioIoiOiioioioioiioioIoiioioiOiioioioiIoioioiioioioiiOioIoiioioioiioioioiIoiOioiioioioiioioIoiioioiOiioioioiIoioioioiioioiOiioIoiioioioioiioioioIiOioioiioioioiioI
Lifetime:
Rx from publisher: /192.168.0.21:10000
             elapsed: 1235ms
         packet size: 1468
          throughput: 113 MB/sec
                      80972 packets/sec
            received: 100000 of 100000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.338602ms; acks 19992


Now:
Rx from publisher: /192.168.0.21:10000
             elapsed: 1235ms
         packet size: 1468
          throughput: 113 MB/sec
                      80972 packets/sec
            received: 100000 of 100000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.339186ms; acks 19992
 
oioiioiOioiioioioIioioioiioioiOiioIoioioiioioioiioioIO
Tx summary 1 peers:
   life: 14 MB/sec, 10239 packets/sec
    now: 14 MB/sec, 10239 packets/sec
iioioiioioioioiIoioioiiOioioiioioIoiioioioiioiOiioIoioiioioioioiioioIOiioioioiioioioiIoioioiiOioioiioioIoiioioioiioiOioioI
Lifetime:
Rx from publisher: /192.168.0.21:10000
             elapsed: 2471ms
         packet size: 1468
          throughput: 113 MB/sec
                      80939 packets/sec
            received: 200000 of 200000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.287305ms; acks 37107


Now:
Rx from publisher: /192.168.0.21:10000
             elapsed: 1234ms
         packet size: 1468
          throughput: 113 MB/sec
                      81037 packets/sec
            received: 100000 of 100000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.227671ms; acks 17115
 
ioioioioiioioioiOIoiioioioioiioioioIoOoooooooooOoooooooooOoooooooooO
Tx summary 1 peers:
   life: 25 MB/sec, 17691 packets/sec
    now: 91 MB/sec, 65020 packets/sec
oooooooooOoooooooooOoooooooooO^C
 
Applicable Versions
  • Oracle Coherence 11g (11.1.1.9.0)

 

Sunday, November 26, 2017

MulticastUdpSocket failed to set receive buffer size in SOA Suite 11.1.1.9 cluster

Problem

The following appears to be repeating every 75 seconds exactly in soa_server1-diagnostic.log:
[2017-03-15T13:02:33.077+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:02:33.077/235.749 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance. 
[2017-03-15T13:03:48.114+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:03:48.114/310.786 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance. 
[2017-03-15T13:05:03.182+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:05:03.182/385.854 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.  
[2017-03-15T13:06:18.221+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:06:18.221/460.893 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance. 
[2017-03-15T13:07:33.245+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:07:33.245/535.917 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
Solution

1. As 'root', edit /etc/sysctl.conf and add the following:
net.core.rmem_max=2096304
net.core.wmem_max=2096304
net.core.rmem_max=4194304                                                      
net.core.wmem_max=4194304
2. Bounce the SOA managed servers.

References
 
Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)
  • Oracle Coherence 11g (11.1.1.9.0)