Setting the ulimit for Oracle SOA Suite 11g

Problem:

Oracle SOA Suite 11g became temporarily unavailable under heavy load. Neither SOA Suite nor WebLogic Server crashed at the time, but soon after became accessible.

Upon further investigation, the errors found in /var/log/messages was (where 10000 was the uid of the 'oracle' unix user):

Aug 31 20:53:22 soadev sshd[22480]: fatal: setresuid 10000: Resource temporarily unavailable
Aug 31 20:53:33 soadev sshd[22597]: fatal: setresuid 10000: Resource temporarily unavailable
Aug 31 20:53:34 soadev sshd[22612]: fatal: setresuid 10000: Resource temporarily unavailable
Aug 31 20:53:34 soadev sshd[22619]: fatal: setresuid 10000: Resource temporarily unavailable

Upon trying to SSH into the linux server, the following was received on the SSH client:

Last login: Wed Aug 31 20:06:49 2011 from somedomain.somedomain
-bash: fork: Resource temporarily unavailable
-bash-3.2$

It appeared that the OS limits may be set a tad too low:

oracle@soadev:/home/oracle> ulimit -p -n -i -u
pipe size (512 bytes, -p) 8
open files (-n) 4096
pending signals (-i) 163904
max user processes (-u) 2047

This may contribute to an OOM exception observed in the SOA server several minutes later:

####<Aug 31, 2011 8:59:41 PM EDT> <Critical> <WorkManager> <soadev> <soa_server2> <[STANDBY] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <b9cc3a64b01a130f:-6178527a:1322258bdd6:-8000-0000000000010b4c> <1314838781501> <BEA-002911> <WorkManager weblogic.jms.MyJMSServer_2.System failed to schedule a request due to java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Java heap 6G reserved, 6G committed
Paged memory=18014398500568380K/36425712K.
Your Java heap size might be set too high.
Try to reduce the Java heap size using -Xmx:<size> (e.g. "-
java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Java heap 6G reserved, 6G committed
Paged memory=18014398500568380K/36425712K.
Your Java heap size might be set too high.
Try to reduce the Java heap size using -Xmx:<size> (e.g. "-

Solution:

The OS limits may be set low. Consider doing the following.

1. Add the following to /etc/sysctl.conf:

kernel.shmmni = 4096

kernel.sem = 256 32000 100 142

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 4194304

net.core.rmem_max = 4194304

net.core.wmem_default = 4194304

net.core.wmem_max = 4194304

fs.file-max = 134283264

fs.aio-max-nr = 1048576

vm.min_free_kbytes = 512000

2. Add the following to /etc/security/limits.conf:

oracle soft nproc 16384

oracle hard nproc 63536

oracle soft nofile 16384

oracle hard nofile 63536

3. Add the following to /etc/pam.d/login:

session required /lib/security/pam_limits.so

4. Add the following to /etc/profile (for 32-bit Linux):

if [ $USER = "oracle" ]; then

  if [ $SHELL = "/bin/ksh" ]; then

    ulimit -p 16384

    ulimit -n 65536

  else

    ulimit -u 16384 -n 65536

  fi

fi

    Add the following to /etc/profile (for 64-bit Linux):

if [ $USER = "oracle" ]; then

  if [ $SHELL = "/bin/ksh" ]; then

    ulimit -Su 16383

    ulimit -Hu 16383

    ulimit -Sn 63535

    ulimit -Hn 63535

  else

    ulimit -Hn 63535 -Sn 63535 -Hu 16383 -Su 16383

  fi

fi

Applicable Versions:

• Oracle SOA Suite 11g (11.1.1.x) on Red Hat 5.5 (32-bit & 64-bit)

References:

• http://blog.ipnweb.com/2010/09/no-nonsense-installation-of-oracle.html