BEA-381304: SocketMuxer detected socket closure while waiting for a response

Problem:

We received the following error in the osb_server1.log file during a long running OSB service (the pipeline had a loop making hundreds of callouts, for each one writing to a JMS queue).

<Feb 27, 2015 1:04:18 AM EST> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: SocketMuxer detected socket closure while waiting for a response
java.io.IOException: SocketMuxer detected socket closure while waiting for a response
        at weblogic.net.http.SocketClosedNotification.<clinit>(SocketClosedNotification.java:10)
        at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse.handleError(AsyncResponseHandler.java:396)
        at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse.hasException(AsyncResponseHandler.java:504)
        at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:827)
        at weblogic.socket.SocketMuxer.deliverHasException(SocketMuxer.java:767)
        Truncated. see log file for complete stacktrace

Solution:

Note that if the Proxy and the Business Service both use the same Work Manager then there is potential for starvation.

For example:

• Request Pipeline makes a blocking callout, say to perform a database read.
• Business Service response tries to allocate a thread from thread pool but all threads are blocked in the database read.
• New requests arrive and contend with responses arriving for the available threads.
• Similar problems can occur if the response pipeline blocks for some reason, maybe a database update for example.

The solution to this is to make sure that the Proxy and Business Service use different Work Managers so that they do not contend with each other for threads.


References:

• https://blogs.oracle.com/reynolds/entry/following_the_thread_in_osb

Applicable Versions:

• Oracle Service Bus (OSB) 11g