BEA-000337 and STUCK thread in OSB 11g

Problem

Seeing the AdminServer in warning state and a stuck thread. AdminServer.log shows this:

####<Dec 15, 2018 4:54:00 PM GMT> <Error> <WebLogicServer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004fb0> <1489596840017> <BEA-000337> <[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,433" seconds working on the request "Workmanager: weblogic.kernel.Default, Version: 0, Scheduled=false, Started=true, Started time: 1434264 ms
", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
null>

Solution

1. Look through the AdminServer.log log file. Could be related to an OutOfMemory exception that occurred during deployment.

####<Dec 15, 2018 4:32:00 PM GMT> <Warning> <Deployer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004f91> <1489595520836> <BEA-149078> <Stack trace for message 149004
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.notifyPrepareFailure(ServerDeploymentReceiver.java:355)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver._prepare(ServerDeploymentReceiver.java:434)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.access$700(ServerDeploymentReceiver.java:70)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver$1.run(ServerDeploymentReceiver.java:246)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.prepare(ServerDeploymentReceiver.java:213)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.lang.String.substring(String.java:1877)

Applicable Versions

• Oracle Service Bus (OSB) 11g (11.1.1.9.0)

 

HTTP 500 when logging in to Siebel with SSO/X509

Problem

When accessing the Siebel application at: https://siebelhost1.raastech.com/epublicsector_enu/, you get an HTTP 500 on the browser.

All services are up.

Nothing is in any of these logs:

• WebGate --> C:\NetPoint\Webgate\access\oblix\logs\oblog.log
• Siebel --> C:\Siebel\15.0.0.0.0\eappweb\log\ss170601_2516.log
• IIS --> C:\inetpub\logs\LogFiles\W3SVC1\u_ex170606.log

Solution

1. There was likely an IAM outage. You will need to restart the SIE01 and SIE02 webtiers.

Applicable Versions

• Oracle Access Manager 11g

 

BEA-000403 MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds

Problem

Error received during OSB deployment:

<Jan 2, 2018 7:13:01 PM GMT> <Error> <Socket> <BEA-000403> <IOException occurred on socket: Socket[addr=osbhost1.raastech.com/192.168.0.30,port=7001,localport=57684]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes for protocol: 't3'
        at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
        at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:369)
        at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:984)
        at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
        at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:887)
        at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:343)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
> 

Solution

1. Navigate to Servers > AdminServer > Protocols and "Enable Tunneling"

2. Make this change to the envcode.properties file (or equivalent) when deploying (basically, use HTTP instead of T3):

OLD: osbAdminUrl=t3://192.168.0.30:7001

NEW: osbAdminUrl=http://192.168.0.30:7001

Applicable Versions

• Oracle WebLogic Server 11g (11.1.1.9.0)
• Oracle Service Bus (OSB) 11g (11.1.1.9.0)