BEA-000337 and STUCK thread in OSB 11g

Problem

Seeing the AdminServer in warning state and a stuck thread. AdminServer.log shows this:

####<Dec 15, 2018 4:54:00 PM GMT> <Error> <WebLogicServer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004fb0> <1489596840017> <BEA-000337> <[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,433" seconds working on the request "Workmanager: weblogic.kernel.Default, Version: 0, Scheduled=false, Started=true, Started time: 1434264 ms
", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds. Stack trace:
null>

Solution

1. Look through the AdminServer.log log file. Could be related to an OutOfMemory exception that occurred during deployment.

####<Dec 15, 2018 4:32:00 PM GMT> <Warning> <Deployer> <osbhost1.raastech.com> <AdminServer> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <dd3937adc7e76694:5ea517f4:15ac77c3297:-8000-0000000000004f91> <1489595520836> <BEA-149078> <Stack trace for message 149004
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.notifyPrepareFailure(ServerDeploymentReceiver.java:355)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver._prepare(ServerDeploymentReceiver.java:434)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.access$700(ServerDeploymentReceiver.java:70)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver$1.run(ServerDeploymentReceiver.java:246)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.prepare(ServerDeploymentReceiver.java:213)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.lang.String.substring(String.java:1877)

Applicable Versions

• Oracle Service Bus (OSB) 11g (11.1.1.9.0)