Tuesday, April 24, 2018

ORABPEL-9732 'Persistency service internal error: Installation failed' during startup

Problem

During the startup of the SOA managed servers, you see the following ORABPEL-9732 error and multiple "Installation failed" errors:
<Jun 1, 2017 3:30:00 PM GMT> <Notice> <Stdout> <BEA-000000> <<Jun 1, 2017 3:30:00 PM GMT> <Error> <oracle.soa.services.workflow.common> <BEA-000000> <<.> Persistency service internal error: Persistency service internal error: Installation failed.
Persistency service experienced unexpected internal error.
Check the underlying exception and correct the error. If the error persists, contact Oracle Support Services.
Persistency service experienced unexpected internal error.
Check the underlying exception and correct the error. If the error persists, contact Oracle Support Services.
      
ORABPEL-9732
Solution

1. Restore the database.

There is a problem with the database. It could be empty tables, missing sequences, or invalid objects. Don't waste your time troubleshooting and restore the database from a working copy.


Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Thursday, April 5, 2018

BEA-000297 Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate

Problem

You try to configure SSL on the managed server, but it is not working and you get the following error in the log:
<Jun 18, 2017 5:57:38 PM GMT> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias soahost1.raastech.com from the JKS keystore file /u01/app/oracle/middleware/keystores/raastech.jks.>
<Jun 18, 2017 5:57:38 PM GMT> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>
Solution

1. Navigate to [managed server] > Configuration > SSL > Advanced.

2. Check "Use JSSE SSL".

3. Save and Activate changes.

 
Applicable Versions
  • Oracle WebLogic Server 11g (11.1.1.9.0)

 

Sunday, March 25, 2018

Cannot resolve reference to bean 'DOStoreFactory' during SOA startup

Problem

When starting up "soa_server1" it comes up fine. But when we start "soa_server2", we get these errors in the soa_server2.out log file:
<May 2, 2017 1:39:56 PM GMT> <Warning> <HTTP> <BEA-101162> <User defined listener com.oracle.weblogic.wsee.wrapper.org.springframework.web.context.ContextLoaderListener failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SensorManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricMesh' while setting bean property 'fabricMesh'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMesh': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricMeshTarget' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'WSServiceBC' while setting bean property 'bindingComponents' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'WSServiceBC' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'InstanceManager' while setting bean property 'instanceManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'InstanceManager' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'DOStoreFactory' while setting bean property 'DOStoreFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DOStoreFactory' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'FabricConfigManager' while setting bean property 'fabricConfigManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FabricConfigManager' defined in ServletContext resource [/WEB-INF/fabric-config.xml]: Cannot resolve reference to bean 'EventBusConfigurable' while setting bean property 'configurables' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'EventBus' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean 'ClusterInterface' while setting bean property 'clusterInterface'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ClusterInterface' defined in ServletContext resource [/WEB-INF/fabric-config-core.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
 MemberSet=MasterMemberSet(
 ThisMember=Member(Id=2, Timestamp=2017-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
 OldestMember=Member(Id=1, Timestamp=2017-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
 ActualMemberSet=MemberSet(Size=2
 Member(Id=1, Timestamp=2017-05-02 13:34:29.559, Address=192.168.0.21:8088, MachineId=29282, Location=site:,machine:soahost2,process:24971, Role=WeblogicServer)
 Member(Id=2, Timestamp=2017-05-02 13:34:56.129, Address=192.168.0.20:8088, MachineId=23265, Location=site:,machine:soahost1,process:32418, Role=WeblogicServer)
 )
 MemberId|ServiceVersion|ServiceJoined|MemberState
 1|3.7.1|2017-05-02 13:34:29.559|JOINING,
 2|3.7.1|2017-05-02 13:34:56.523|JOINED
 RecycleMillis=1200000
 RecycleSet=MemberSet(Size=0
 )
 )
 ).
You may also see this error:
<May 2, 2017 1:41:26 PM GMT> <Error> <Cluster> <BEA-000140> <Failed to deserialize statedump from server 6011237482029920459S:192.168.0.21:[8001,8001,-1,-1,-1,-1,-1]:soahost1:8001,soahost2:8001:soadomain:soa_server2 with java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote.
 java.lang.ClassNotFoundException: oracle.bpm.bpmn.engine.instancemanagement.ejb.UserInitiateProcessCleanupRemote
 at weblogic.application.internal.AppClassLoaderManagerImpl.loadApplicationClass(AppClassLoaderManagerImpl.java:164)
 at weblogic.common.internal.ProxyClassResolver.resolveProxyClass(ProxyClassResolver.java:68)
 at weblogic.common.internal.WLObjectInputStream.resolveProxyClass(WLObjectInputStream.java:76)
 at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
 Truncated. see log file for complete stacktrace
 >
Even if you have opened up all relevant ports, the problem persists:
root@soahost1> iptables -S | grep IN_public_allow
 -N IN_public_allow
 -A IN_public -j IN_public_allow
 -A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 7001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 8001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 9001 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 5556 -m conntrack --ctstate NEW -j ACCEPT
 -A IN_public_allow -p tcp -m tcp --dport 7 -m conntrack --ctstate NEW -j ACCEPT
Solution

1. Turn off the local firewall:
service firewalld stop
You may need to spend more time to open up UDP and multicast on every NIC to re-enable the local firewall.

 
References
 
Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Tuesday, March 20, 2018

Getting java.lang.ExceptionInInitializerError during OSB deployment

Problem

During the deployment of the OSB project from the command line, we get the following cryptic error:
Diagnostic XML Bean debug log file created: /tmp/xmlbeandebug5401068355720061527.log
* Unexpected error: java.lang.ExceptionInInitializerError
* Session discarded
* Unexpected error:  java.lang.ExceptionInInitializerError
No stack trace available.

Problem invoking WLST - Traceback (innermost last):
  File "/home/oracle/deploy/osb_import.py", line 277, in ?
  File "/home/oracle/deploy/osb_import.py", line 134, in importToALSBDomain
        at sun.misc.Unsafe.ensureClassInitialized(Native Method)
        at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
        at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:140)
        at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1057)
        at java.lang.reflect.Field.getFieldAccessor(Field.java:1038)
        .
        .
        .
java.lang.ExceptionInInitializerError: java.lang.ExceptionInInitializerError
Opening up the .log file in /tmp doesn't help much and only yields more cryptic information:
XML-BEANS compiled schema: Could not locate compiled schema resource schemaorg_apache_xmlbeans/system/resources_schema_xbtypes/index.xsb (schemaorg_apache_xmlbeans.system.resources_schema_xbtypes.index) - code 0
org.apache.xmlbeans.SchemaTypeLoaderException: XML-BEANS compiled schema: Could not locate compiled schema resource schemaorg_apache_xmlbeans/system/resources_schema_xbtypes/index.xsb (schemaorg_apache_xmlbeans.system.resources_schema_xbtypes.index) - code 0
        at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl$XsbReader.<init>(SchemaTypeSystemImpl.java:1519)
        at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl.initFromHeader(SchemaTypeSystemImpl.java:273)
        at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl.<init>(SchemaTypeSystemImpl.java:185)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

Solution

1. Try importing the JAR through the OSB Console, and you will get a more accurate error as to why the deployment failed.

Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)


Friday, March 9, 2018

BEA-473003 Destination unreachable on t3s; nested exception is: java.net.NoRouteToHostException

Problem

Seeing these errors in osb_server1.out log file (note the T3S protocol in the log):
<Mar 29, 2017 4:14:00 PM GMT> <Error> <ALSB Statistics Manager> <BEA-473003> <Aggregation Server Not Available. Failed to get remote aggregator
javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://192.168.0.30:8011: Destination unreachable; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); No available router to destination]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:368)
        at weblogic.jndi.Environment.getContext(Environment.java:315)
        at weblogic.jndi.Environment.getContext(Environment.java:285)
        Truncated. see log file for complete stacktrace
Caused By: java.net.ConnectException: t3s://192.168.0.30:8015: Destination unreachable; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); No available router to destination
        at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
        at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
        at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:353)
        at weblogic.jndi.Environment.getContext(Environment.java:315)
        Truncated. see log file for complete stacktrace
Caused By: java.rmi.ConnectException: Destination unreachable; nested exception is:
        java.net.NoRouteToHostException: No route to host (Host unreachable); No available router to destination
        at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)
        at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321)
        at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:262)
        at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:199)
        at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
        Truncated. see log file for complete stacktrace
>
Solution

1. Log in to the WebLogic Admin Console

2. Navigate to [managed server] > SSL > Advanced

3. Set "Hostname Verification" to "None"

4. Repeat for all managed servers

5. Restart the managed servers


Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Sunday, February 18, 2018

BEA-423223 There was an error while running a lifecycle stage

Problem

When deploying an OSB project, you may get the following undescriptive error on the OSB Console:
An unexpected error occurred.
The following error would appear in the AdminServer.out file:
####<Mar 3, 2017 5:03:21 PM GMT> <Error> <netuix> <osbhost1> <AdminServer> <[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <685ea1de428af539:-5fcff8e9:82f2515a61f:-8000-0000000000003938> <1488560601009> <BEA-423137> <There was an error loading the requested URI null.> 
####<Mar 3, 2017 5:03:21 PM GMT> <Error> <netuix> <osbhost1> <AdminServer> <[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <de428af5685ea139:-5fcff8e9:15a61f82f25:-8000-0000000000003938> <1488560601009> <BEA-423223> <There was an error while running a lifecycle stage :: Lifecycle: UIControl.render :: for the control :: null ::.
com.bea.netuix.nf.UIControlException: com.bea.portlet.adapter.scopedcontent.ActionLookupFailedException: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.readFully(Ljava/io/InputStream;[B)V
        at com.bea.netuix.servlets.controls.content.NetuiContent.checkPreRenderExceptions(NetuiContent.java:404)
        at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:343)
        at com.bea.netuix.nf.ControlLifecycle$7.visit(ControlLifecycle.java:485)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:518)
Solution

1. The OSB JAR is missing the ExportInfo file. The developer must fix this..


Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Thursday, February 15, 2018

BEA-149205 Failed to initialize the application '_ALSB'

Problem

Getting the following error when starting up the OSB managed server. Multiple applications are failing, and this is just a snippet of one of the errors:
<Jun 30, 2017 12:38:45 PM GMT> <Error> <Deployer> <BEA-149205> <Failed to initialize the application '_ALSB_1491592089579' due to error weblogic.management.DeploymentException: Exception occured while downloading files.
weblogic.management.DeploymentException: Exception occured while downloading files
        at weblogic.deploy.internal.targetserver.datamanagement.AppDataUpdate.doDownload(AppDataUpdate.java:43)
        at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
        at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:97)
        at weblogic.deploy.internal.targetserver.BasicDeployment.prepareDataUpdate(BasicDeployment.java:704)
        at weblogic.deploy.internal.targetserver.BasicDeployment.stageFilesForStatic(BasicDeployment.java:747)
        Truncated. see log file for complete stacktrace
Caused By: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from admin server for deployment request "0". Underlying error is: "null"
        at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:86)
        at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:165)
        at weblogic.deploy.internal.targetserver.datamanagement.AppDataUpdate.doDownload(AppDataUpdate.java:39)
        at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
        at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:97)
        Truncated. see log file for complete stacktrace
>
 
Solution

1. You must undeploy these applications

2. Start up the AdminServer

3. Connect through WLST and run the following. Make sure to undeploy the applications that are showing up in the logs, so do not use the same exact values here.
cd /u01/app/oracle/middleware/wlserver_10.3/common/bin/ 
./wlst.sh 
connect( 'weblogic', 'welcome1', 't3://localhost:7001') 
stopApplication('_ALSB_1492546342692') 
undeploy('_ALSB_1492546342692')  
stopApplication('_ALSB_1492545743924')
undeploy('_ALSB_1492545743924')
stopApplication('_ALSB_1491592307429')
undeploy('_ALSB_1491592307429')
stopApplication('_ALSB_1491592306429')
undeploy('_ALSB_1491592306429')
stopApplication('_ALSB_1491592089579')
undeploy('_ALSB_1491592089579')
disconnect()
4. Login to WebLogic Admin Console and activate the session
 
 
References
 
Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Wednesday, January 17, 2018

BEA-000337 and STUCK thread in OSB 11g

Problem

Seeing the AdminServer in warning state and a stuck thread. AdminServer.log shows this:
####<Mar 15, 2017 4:54:00 PM GMT> <Error> <WebLogicServer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004fb0> <1489596840017> <BEA-000337> <[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,433" seconds working on the request "Workmanager: weblogic.kernel.Default, Version: 0, Scheduled=false, Started=true, Started time: 1434264 ms
", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
null>
Solution

1. Look through the AdminServer.log log file. Could be related to an OutOfMemory exception that occurred during deployment.
####<Mar 15, 2017 4:32:00 PM GMT> <Warning> <Deployer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004f91> <1489595520836> <BEA-149078> <Stack trace for message 149004
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.notifyPrepareFailure(ServerDeploymentReceiver.java:355)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver._prepare(ServerDeploymentReceiver.java:434)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.access$700(ServerDeploymentReceiver.java:70)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver$1.run(ServerDeploymentReceiver.java:246)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.prepare(ServerDeploymentReceiver.java:213)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.lang.String.substring(String.java:1877)

Applicable Versions
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Friday, January 5, 2018

HTTP 500 when logging in to Siebel with SSO/X509

Problem

When accessing the Siebel application at: https://siebelhost1.raastech.com/epublicsector_enu/, you get an HTTP 500 on the browser.
 
All services are up.
 
Nothing is in any of these logs:
  • WebGate --> C:\NetPoint\Webgate\access\oblix\logs\oblog.log
  • Siebel --> C:\Siebel\15.0.0.0.0\eappweb\log\ss170601_2516.log
  • IIS --> C:\inetpub\logs\LogFiles\W3SVC1\u_ex170606.log
 
Solution

1. There was likely an IAM outage. You will need to restart the SIE01 and SIE02 webtiers.


Applicable Versions
  • Oracle Access Manager 11g

 

Monday, January 1, 2018

BEA-000403 MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds

Problem

Error received during OSB deployment:
<May 2, 2017 7:13:01 PM GMT> <Error> <Socket> <BEA-000403> <IOException occurred on socket: Socket[addr=osbhost1.raastech.com/192.168.0.30,port=7001,localport=57684]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
        at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
        at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:369)
        at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:984)
        at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
        at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:887)
        at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:343)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
>
 
Solution

1. Navigate to Servers > AdminServer > Protocols and "Enable Tunneling"

2. Make this change to the envcode.properties file (or equivalent) when deploying (basically, use HTTP instead of T3):
OLD: osbAdminUrl=t3://192.168.0.30:7001

NEW: osbAdminUrl=http://192.168.0.30:7001

Applicable Versions
  • Oracle WebLogic Server 11g (11.1.1.9.0)
  • Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

Tuesday, December 26, 2017

Cloning OID 11g

Oracle does not support the changing of hostname or IP address for your Oracle Internet Directory 11g installation, rendering cloning impossible. These instructions, however, guide you through that process.


1. Set the environment:
export ORACLE_HOME=/u01/app/oracle/middleware/Oracle_OID
export ORACLE_INSTANCE=/u01/app/oracle/middleware/asinst_1
export TNS_ADMIN=$ORACLE_INSTANCE/config 
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/ldap/bin:$ORACLE_INSTANCE/bin:$PATH
2. Clear the temp folders on node 1 (for AdminServer, wls_ods1):
cd ${ASERVER_HOME}/servers/AdminServer
rm -rf cache/* logs/* tmp/*
cd ${MSERVER_HOME}/servers/wls_ods1
rm -rf cache/* logs/* tmp/* data/*
cd ${MSERVER_HOME}/servers/wls_ods2 
rm -rf cache/* logs/* tmp/* data/*
3. Replace values in red:
cd /u01/app/oracle/middleware
find . -type f \( ! -name "oidpwdr*" ! -name "*.DAT" ! -name "*.cer" ! -name "*dump1" ! -name "*.swp" ! -name "*.jks" \) -exec sed -i "s/OLDHOST/NEWHOST/g" {} \;
find . -type f \( ! -name "oidpwdr*" ! -name "*.DAT" ! -name "*.cer" ! -name "*dump1" ! -name "*.swp" ! -name "*.jks" \) -exec sed -i "s/OLDHOST/NEWHOST/g" {} \; 
find . -type f \( ! -name "oidpwdr*" ! -name "*.DAT" ! -name "*.cer" ! -name "*dump1" ! -name "*.swp" ! -name "*.jks" \) -exec sed -i "s/.olddomain.com/.newdomain.com/g" {} \;
4. Change the strings to the new database string:
FILE: /u01/app/oracle/middleware/asinst_1/config/tnsnames.ora
OLD: OIDDB=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=prod-cluster)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=oiddb.raastech.com)))
FILE: /u01/app/oracle/middleware/user_projects/domains/oiddomain/config/jdbc/schedulerDS-*-jdbc.xml
OLD: <url>jdbc:oracle:thin:@//prod-cluster:1521/oiddb.raastech.com</url>
OLD: <value>1521</value>
OLD: <value>prod-cluster</value>
5. Run the change-ip-host script:
nohup ${ASERVER_HOME}/bin/startWebLogic.sh >> ${ASERVER_HOME}/servers/${ADMINSERVER}/logs/${ADMINSERVER}.out & 
cd $ORACLE_HOME 
$ORACLE_HOME/chgip/scripts/chgiphost.sh -oldhost OLDHOST -newhost NEWHOST -instanceHome /u01/app/oracle/middleware/asinst_1 
$ORACLE_INSTANCE/bin/opmnctl unregisterinstance -instanceName asinst_1 -adminHost NEWHOST.raastech.com -adminPort 7001 
$ORACLE_INSTANCE/bin/opmnctl redeploy -adminHost NEWHOST.raastech.com -adminPort 7001 
ll /u01/app/oracle/middleware/user_projects/domains/oiddomain/servers/AdminServer/upload/NonJ2EEManagement/11.1.1/app/NonJ2EEManagement.ear 
$ORACLE_INSTANCE/bin/opmnctl redeploy -adminHost NEWHOST.raastech.com -adminPort 7001

$ORACLE_INSTANCE/bin/opmnctl registerinstance -adminHost NEWHOST.raastech.com -adminPort 7001
If the 'redeploy' or 'registerinstance' actions fail, do not worry. Just confirm that "NonJ2EEManagement" is a deployed application and exists in /u01/app/oracle/middleware/user_projects/domains/oiddomain/servers/AdminServer/upload/NonJ2EEManagement/11.1.1/app/NonJ2EEManagement.ear.
 6. Remove the states folder to avoid the "Failed to open locale state file" error:
rm -rf /u01/app/oracle/middleware/asinst_1/config/OPMN/states
7. Reset the ODS password in the wallet (make sure this password matches that of the ODS user in the database):
cd $ORACLE_INSTANCE/OID/admin

mv oidpwdlldap1 oidpwdlldap1.old

$ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB create_wallet=true
8. Startup the rest of the environment:
nohup ${MSERVER_HOME}/bin/startManagedWebLogic.sh wls_ods1 >> ${MSERVER_HOME}/servers/wls_ods1/logs/wls_ods1.out & 
${MW_HOME}/asinst_1/bin/opmnctl startall
9. Verify the installation:
$ORACLE_HOME/bin/ldapbind -h oidhost1.raastech.com -p 3060 -D cn=orcladmin -w welcome1
$ORACLE_HOME/bin/ldapbind -h oidhost1.raastech.com -p 3131 -D cn=orcladmin -w welcome1 -U 1 
$ORACLE_HOME/bin/ldapsearch -h oidhost1.raastech.com -p 3131 -U 1 -D cn=orcladmin -s sub "(objectclass=*)" -w welcome1

Applicable Versions
  • Oracle Internet Directory (OID) 11g (11.1.1.9.0)

 

Wednesday, December 13, 2017

Unable to Connect to Database: Incorrect location for tnsnames.ora in OID 11g

Problem

If you are trying to execute the oidpasswd command, for example:
$ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB create_wallet=true 
$ORACLE_HOME/ldap/bin/oidpasswd connect=OIDDB change_oiddb_pwd=true
You receive the following error on the prompt:
Unable to Connect to Database: Incorrect location for tnsnames.ora (derived from ORACLE_INSTANCE) or Incorrect TNS Connect string or Invalid Password

Solution

1. This is because you must set the TNS_ADMIN environment variable:
export ORACLE_HOME=/u01/app/oracle/middleware/Oracle_OID 
export ORACLE_INSTANCE=/u01/app/oracle/middleware/asinst_1 
export TNS_ADMIN=$ORACLE_INSTANCE/config 
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/ldap/bin:$ORACLE_INSTANCE/bin:$PATH
2. If this does not work, try the following:
cp $TNS_ADMIN/tnsnames.ora $ORACLE_HOME/network/admin
3. If this does not work, the problem is with the "ods" account on the database.
a. Try to connect to the "ods" user using Oracle SQL Developer.

b. Try to connect to the "ods" user using SQL*Plus on the OAM server:  
sqlplus ods/welcome1@oiddb
c. Then try resetting the password again or create a new wallet

 
Applicable Versions
  • Oracle Internet Directory (OID) 11g

 

Saturday, December 2, 2017

Test Coherence communication between nodes using 'multicast-test.sh'

This is a successful execution of the multicast-test.sh script. Any other response may indicate communication issues between the nodes.
oracle@soahost1:/u01/app/oracle/middleware/coherence_3.7/bin> ./multicast-test.sh -local soahost1 
java version "1.7.0_121"
Java(TM) SE Runtime Environment (build 1.7.0_121-b32)
Java HotSpot(TM) 64-Bit Server VM (build 24.121-b32, mixed mode)
 
2017-05-03 19:21:53.805/0.380 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence.xml"
2017-05-03 19:21:53.878/0.453 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2017-05-03 19:21:53.878/0.453 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2017-05-03 19:21:53.881/0.456 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
 
Oracle Coherence Version 3.7.1.1 Build 28901
 Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
 
Starting test on ip=soahost1/192.168.0.20, group=/237.0.0.1:9000, ttl=4
Configuring multicast socket...
Starting listener...
Wed May 03 19:21:53 GMT 2017: Sent packet 1 containing 1468 bytes.
Wed May 03 19:21:54 GMT 2017: Received test packet 1 from self (sent 7ms ago).
Wed May 03 19:21:54 GMT 2017: Received test packet 3 from ip=soahost2/192.168.0.21, group=/237.0.0.1:9000, ttl=4 containing 1468 bytes.
Wed May 03 19:21:56 GMT 2017: Sent packet 2 containing 1468 bytes.
Wed May 03 19:21:56 GMT 2017: Received test packet 2 from self (sent 1ms ago).
Wed May 03 19:21:56 GMT 2017: Received test packet 4 from ip=soahost2/192.168.0.21, group=/237.0.0.1:9000, ttl=4 containing 1468 bytes.
Wed May 03 19:21:58 GMT 2017: Sent packet 3 containing 1468 bytes.
Wed May 03 19:21:58 GMT 2017: Received test packet 3 from self (sent 3ms ago).
Wed May 03 19:21:58 GMT 2017: Received test packet 5 from ip=soahost2/192.168.0.21, group=/237.0.0.1:9000, ttl=4 containing 1468 bytes.
Wed May 03 19:22:00 GMT 2017: Sent packet 4 containing 1468 bytes.
Wed May 03 19:22:00 GMT 2017: Received test packet 4 from self (sent 1ms ago).
Wed May 03 19:22:02 GMT 2017: Sent packet 5 containing 1468 bytes.
Wed May 03 19:22:02 GMT 2017: Received test packet 5 from self (sent 1ms ago).
 
Applicable Versions
  • Oracle Coherence 11g (11.1.1.9.0)

 

Friday, December 1, 2017

Test Coherence communication between nodes using 'datagram-test.sh'

This is a successful execution of the datagram-test.sh script. Any other response may indicate communication issues between the nodes.
oracle@soahost1:/u01/app/oracle/middleware/coherence_3.7/bin> ./datagram-test.sh -polite soahost2 
java version "1.7.0_121"
Java(TM) SE Runtime Environment (build 1.7.0_121-b32)
Java HotSpot(TM) 64-Bit Server VM (build 24.121-b32, mixed mode)
2017-05-03 19:19:37.342/0.319 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence.xml"
2017-05-03 19:19:37.412/0.389 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/u01/app/oracle/middleware/coherence_3.7/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2017-05-03 19:19:37.412/0.389 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2017-05-03 19:19:37.415/0.392 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
 
Oracle Coherence Version 3.7.1.1 Build 28901
 Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
 
creating datagram socket using provider: SystemSocketProvider
starting listener: at /192.168.0.20:9999
packet size: 1468 bytes
buffer size: 1428 packets
  report on: 100000 packets, 139 MBs
    process: 20 bytes/packet
        log: null
     log on: 139 MBs
 
starting publisher: at /192.168.0.20:10000 sending to soahost2/192.168.0.21:9999
packet size: 1468 bytes
buffer size: 32 packets
  report on: 100000 packets, 139 MBs
    process: 20 bytes/packet
      peers: 1
       rate: no limit
 
no packet burst limit
waiting for listener to be contacted before publishing
 
Recieving data from 1 publisher(s).
ioioioioioioiioioIoiOiioioioioiioioIoiioioiOiioioioiIoioioiioioioiiOioIoiioioioiioioioiIoiOioiioioioiioioIoiioioiOiioioioiIoioioioiioioiOiioIoiioioioioiioioioIiOioioiioioioiioI
Lifetime:
Rx from publisher: /192.168.0.21:10000
             elapsed: 1235ms
         packet size: 1468
          throughput: 113 MB/sec
                      80972 packets/sec
            received: 100000 of 100000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.338602ms; acks 19992


Now:
Rx from publisher: /192.168.0.21:10000
             elapsed: 1235ms
         packet size: 1468
          throughput: 113 MB/sec
                      80972 packets/sec
            received: 100000 of 100000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.339186ms; acks 19992
 
oioiioiOioiioioioIioioioiioioiOiioIoioioiioioioiioioIO
Tx summary 1 peers:
   life: 14 MB/sec, 10239 packets/sec
    now: 14 MB/sec, 10239 packets/sec
iioioiioioioioiIoioioiiOioioiioioIoiioioioiioiOiioIoioiioioioioiioioIOiioioioiioioioiIoioioiiOioioiioioIoiioioioiioiOioioI
Lifetime:
Rx from publisher: /192.168.0.21:10000
             elapsed: 2471ms
         packet size: 1468
          throughput: 113 MB/sec
                      80939 packets/sec
            received: 200000 of 200000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.287305ms; acks 37107


Now:
Rx from publisher: /192.168.0.21:10000
             elapsed: 1234ms
         packet size: 1468
          throughput: 113 MB/sec
                      81037 packets/sec
            received: 100000 of 100000
             missing: 0
        success rate: 1.0
        out of order: 0
          avg offset: 0
                gaps: 0
        avg gap size: 0
        avg gap time: 0ms
        avg ack time: 1.227671ms; acks 17115
 
ioioioioiioioioiOIoiioioioioiioioioIoOoooooooooOoooooooooOoooooooooO
Tx summary 1 peers:
   life: 25 MB/sec, 17691 packets/sec
    now: 91 MB/sec, 65020 packets/sec
oooooooooOoooooooooOoooooooooO^C
 
Applicable Versions
  • Oracle Coherence 11g (11.1.1.9.0)

 

Sunday, November 26, 2017

MulticastUdpSocket failed to set receive buffer size in SOA Suite 11.1.1.9 cluster

Problem

The following appears to be repeating every 75 seconds exactly in soa_server1-diagnostic.log:
[2017-03-15T13:02:33.077+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:02:33.077/235.749 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance. 
[2017-03-15T13:03:48.114+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:03:48.114/310.786 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance. 
[2017-03-15T13:05:03.182+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:05:03.182/385.854 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.  
[2017-03-15T13:06:18.221+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:06:18.221/460.893 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance. 
[2017-03-15T13:07:33.245+00:00] [soa_server1] [WARNING] [] [Coherence] [tid: Logger@9270387 3.7.1.13] [userId: <anonymous>] [ecid: 0000LfI3B__3b605zzT4iW1OmJeA000003,1:17737] [APP: soa-infra] 2017-03-15 13:07:33.245/535.917 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListenerN, member=1): MulticastUdpSocket failed to set receive buffer size to 16 packets (1023KB); actual size is 20%, 3 packets (208KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
Solution

1. As 'root', edit /etc/sysctl.conf and add the following:
net.core.rmem_max=2096304
net.core.wmem_max=2096304
net.core.rmem_max=4194304                                                      
net.core.wmem_max=4194304
2. Bounce the SOA managed servers.

References
 
Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)
  • Oracle Coherence 11g (11.1.1.9.0)

 

Friday, November 10, 2017

ORA-01017: invalid username/password when starting up OID 11g

Problem

When trying to startup the OID OPMN processes:
${ORACLE_HOME}/asinst_1/bin/opmnctl startall
In this file /u01/app/oracle/middleware/asinst_1/diagnostics/logs/OID/oid1/oidmon-0000.log, you see the following error:
[2017-04-26T21:39:25.10925+00:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oamhost1.raastech.com] [pid: 17029] [tid: 0] Guardian: Connecting to database, connect string is oiddb 
[2017-04-26T21:39:26.57778+00:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oamhost1.raastech.com] [pid: 17029] [tid: 0] Guardian: [gsdsiConnect] ORA-1017, ORA-01017: invalid username/password; logon denied
Solution

1. The DBA should unlock the ODS account:
ALTER USER ods ACCOUNT UNLOCK;
2. Connect as ODS and see if this query works:
CONN ods/**********;

SELECT instance_name FROM v$instance WHERE instance_number IN (SELECT MIN(instance_number) FROM v$instance);
3. If the query returns an ORA-00942 error, then log in as SYSTEM and grant these permissions:
SELECT * FROM sys.dba_role_privs WHERE grantee = "ODS";

ALTER USER ods DEFAULT ROLE SELECT_CATALOG_ROLE, RESOURCE, CONNECT;

Applicable Versions
  • Oracle Internet Directory (OID) 11g

 

Wednesday, November 8, 2017

Coherence WKA configuration changes with 11.1.1.9

With 11.1.1.9, Oracle has changed the instructions on what they recommend for the Coherence configuration for the WKA1 and WKA2 settings.

This is wrong: 

soa_server1
-Dtangosol.coherence.wka1=soahost1 -Dtangosol.coherence.wka2=soahost2 -Dtangosol.coherence.localhost=soahost1 -Dweblogic.MaxMessageSize=50000000 -Dtangosol.coherence.localport=8088 -Dtangosol.coherence.wka1.port=8088 -Dtangosol.coherence.wka2.port=8088

soa_server2
-Dtangosol.coherence.wka1=soahost2 -Dtangosol.coherence.wka2=soahost1 -Dtangosol.coherence.localhost=soahost2 -Dweblogic.MaxMessageSize=50000000 -Dtangosol.coherence.localport=8088 -Dtangosol.coherence.wka1.port=8088 -Dtangosol.coherence.wka2.port=8088

This is right:

soa_server1
-Dtangosol.coherence.wka1=soahost1 -Dtangosol.coherence.wka2=soahost2 -Dtangosol.coherence.localhost=soahost1 -Dweblogic.MaxMessageSize=50000000 -Dtangosol.coherence.localport=8088 -Dtangosol.coherence.wka1.port=8088 -Dtangosol.coherence.wka2.port=8088

soa_server2
-Dtangosol.coherence.wka1=soahost1 -Dtangosol.coherence.wka2=soahost2 -Dtangosol.coherence.localhost=soahost2 -Dweblogic.MaxMessageSize=50000000 -Dtangosol.coherence.localport=8088 -Dtangosol.coherence.wka1.port=8088 -Dtangosol.coherence.wka2.port=8088

 
References
 
Applicable Versions
  • Oracle SOA Suite 11g (11.1.1.9.0)

 

Tuesday, October 10, 2017

JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store

Problem

If you are trying to start up the OAM AdminServer, you see the following error in the logs, then the server dies:
<Jun 20, 2017 6:46:03 PM GMT> <Error> <Security> <BEA-090892> <The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.JpsException: JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store; the target DN must be pre-configured.>
<Jun 20, 2017 6:46:03 PM GMT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.JpsException: JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store; the target DN must be pre-configured.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.JpsException: JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store; the target DN must be pre-configured.
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.JpsException: JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store; the target DN must be pre-configured.
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:190)
        at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:306)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:279)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.JpsException: JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store; the target DN must be pre-configured.
        at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3180)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3480)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3466)
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:188)
        at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.credstore.CredStoreException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.JpsException: JPS-01013: The credential store DN cn=CredentialStore,cn=IAM,cn=JPSContext,cn=jpsroot is missing in the store; the target DN must be pre-configured.
        at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.init(LdapCredentialStore.java:177)
        at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.<init>(LdapCredentialStore.java:166)
        at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.<init>(LdapCredentialStore.java:140)
        at oracle.security.jps.internal.credstore.ldap.LdapCredentialStoreProvider.getInstance(LdapCredentialStoreProvider.java:130)
        at oracle.security.jps.internal.credstore.rdbms.DbmsCredentialStoreProvider.getInstance(DbmsCredentialStoreProvider.java:78)
        Truncated. see log file for complete stacktrace
>
 
<Jun 20, 2017 6:46:03 PM GMT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
Solution

1. The DEV_OPSS schema either has no data, or it is a permission thing.

For example, running SELECT COUNT(1) FROM jps_attrs will return 0 records when there should be millions of records in that table.


References
  • Oracle Doc ID 2066916.1 


Applicable Versions
  • Oracle Access Manager 11g

 

Importing SSL user certificate to an Oracle Wallet

Oracle Wallet does not allow you to import a user certificate if the CSR was not generated in that wallet.

These instructions explain how to take your existing user cert and private key and generate a new Oracle Wallet with them. This is based off the instructions from the Oracle A-Team with some slight variation, so read that note first.

Run these instructions to create the ewallet.p12 file:
export JAVA_HOME=/u01/oracle/java 
export PATH=$JAVA_HOME/bin:$PATH

cat Trust.cer > combined.crt
cat Intermediate.cer >> combined.crt 
cat mycert.cer >> combined.crt

openssl pkcs12 -export -in combined.crt -inkey myprivatekey.key -out ewallet.p12

Afterwards, after running Oracle Wallet Manager, open the directory which contains the ewallet.p12 file (which essentially now becomes your Oracle Wallet directory), click on "Wallet > Auto Login > Exit".


References
 

Tuesday, October 3, 2017

Failed to open locale state file .locale in OID 11g

Problem

In the log file /u01/app/oracle/middleware/asinst_1/diagnostics/logs/OPMN/opmn/opmn.out, you see the following error:
[2017-04-20T09:35:22-04:00] [opmn] [ERROR:1] [752] [pm-internal] Failed to open locale state file /u01/app/oracle/middleware/asinst_1/config/OPMN/opmn/states/.locale (read: No such file or directory)
There are no permission or access issues on the .locale file.


Solution

1. Remove the "states" directory. It will be recreated upon OPMN startup.

 
Applicable Versions
  • Oracle Internet Directory (OID) 11g

 

Friday, September 29, 2017

Apache hardening

Don't forget to add these to your Apache (or OHS) configuration in httpd.conf:
Header set X-XSS-Protection "l;mode=block"Header set Content-Security-Policy "default-src *;"Header set X-Content-Type-Options "nosniff"Header set Cache-Control "no-store"Header set X-Frame-Options SAMEORIGIN

Happy hardening!


Applicable Versions
  • Apache
  • Oracle HTTP Server (OHS) 11g/12c