Saturday, October 1, 2011

Setting the ulimit for Oracle SOA Suite 11g

Problem:

Oracle SOA Suite 11g became temporarily unavailable under heavy load. Neither SOA Suite nor WebLogic Server crashed at the time, but soon after became accessible.

Upon further investigation, the errors found in /var/log/messages was (where 10000 was the uid of the 'oracle' unix user):
Aug 31 20:53:22 soadev sshd[22480]: fatal: setresuid 10000: Resource temporarily unavailable
Aug 31 20:53:33 soadev sshd[22597]: fatal: setresuid 10000: Resource temporarily unavailable
Aug 31 20:53:34 soadev sshd[22612]: fatal: setresuid 10000: Resource temporarily unavailable
Aug 31 20:53:34 soadev sshd[22619]: fatal: setresuid 10000: Resource temporarily unavailable
Upon trying to SSH into the linux server, the following was received on the SSH client:
Last login: Wed Aug 31 20:06:49 2011 from somedomain.somedomain
-bash: fork: Resource temporarily unavailable
-bash-3.2$
It appeared that the OS limits may be set a tad too low:
oracle@soadev:/home/oracle> ulimit -p -n -i -u
pipe size (512 bytes, -p) 8
open files (-n) 4096
pending signals (-i) 163904
max user processes (-u) 2047
This may contribute to an OOM exception observed in the SOA server several minutes later:
####<Aug 31, 2011 8:59:41 PM EDT> <Critical> <WorkManager> <soadev> <soa_server2> <[STANDBY] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <b9cc3a64b01a130f:-6178527a:1322258bdd6:-8000-0000000000010b4c> <1314838781501> <BEA-002911> <WorkManager weblogic.jms.MyJMSServer_2.System failed to schedule a request due to java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Java heap 6G reserved, 6G committed
Paged memory=18014398500568380K/36425712K.
Your Java heap size might be set too high.
Try to reduce the Java heap size using -Xmx:<size> (e.g. "-
java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Java heap 6G reserved, 6G committed
Paged memory=18014398500568380K/36425712K.
Your Java heap size might be set too high.
Try to reduce the Java heap size using -Xmx:<size> (e.g. "-

Solution:

The OS limits may be set low. Consider doing the following.

1. Add the following to /etc/sysctl.conf:
kernel.shmmni = 4096
kernel.sem = 256 32000 100 142
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 4194304
net.core.wmem_max = 4194304
fs.file-max = 134283264
fs.aio-max-nr = 1048576
vm.min_free_kbytes = 512000

2. Add the following to /etc/security/limits.conf:
oracle soft nproc 16384
oracle hard nproc 63536
oracle soft nofile 16384
oracle hard nofile 63536

3. Add the following to /etc/pam.d/login:
session required /lib/security/pam_limits.so

4. Add the following to /etc/profile (for 32-bit Linux):
if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

    Add the following to /etc/profile (for 64-bit Linux):
if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -Su 16383
    ulimit -Hu 16383
    ulimit -Sn 63535
    ulimit -Hn 63535
  else
    ulimit -Hn 63535 -Sn 63535 -Hu 16383 -Su 16383
  fi
fi

Applicable Versions:
  • Oracle SOA Suite 11g (11.1.1.x) on Red Hat 5.5 (32-bit & 64-bit)

References:
  • http://blog.ipnweb.com/2010/09/no-nonsense-installation-of-oracle.html

Ahmed Aboulnaga

No comments: