Friday, April 22, 2011

Performance tuning Oracle SOA Suite 11g

When installed out-of-the-box, Oracle SOA Suite 11g performs adequately for a development environment. In this post, I describe the key areas you need to focus on to improve your performance. I have not done much tuning on the BPEL Process Manager side, so once I do, I will update this blog post.

I have conducted light and heavy load tests on both synchronous and asynchronous Mediator services (not BPEL yet), and my findings and recommendations are documented below.

In summary, here are my findings:
  • Upgrading to PS3 addresses memory instability issues
  • Size your JVM appropriately
  • Moving from Sun JDK to JRockit results in a 32% performance improvement
  • Increasing the Mediator worker threads results in a 30% performance improvement for async services
  • Changing the audit level from Development to Production results in a 46% performance improvement
  • Changing the audit level from Production to Off results in a further 61% performance improvement
  • Tuning the audit configuration causes Production and Off audit levels to perform equally the same
  • Implementing parallel processing of routing rules may improve performance of Mediator anywhere from 4% to 509%
  • I recommend using parallel garbage collection, but don't have statistics


1. Apply the PS3 patchset

I cannot stress this enough. Upgrade to SOA Suite PS3 (11.1.1.4). This resolves an enormous amount of memory issues and will save you a lot of pain.

2. Determine the size needed for your Java Heap Space

If you are running AIA Foundation Pack, then realistically you will need to set your Java Heap Space from 6GB to 10GB, depending on the amount of EBOs you are loading and the amount of composites you have. If you are running just SOA Suite, then you could manage with 2GB to 4GB.

1. Start up your SOA server, and wait until all composites are loaded2. Log in to the WebLogic Admin Console (at http://soaserver:7001/console)
3. Navigate to Home --> Servers --> soa_server1 --> Monitoring --> Performance
4. Click on "Garbage Collect"
5. Inspect the Heap Free Current
6. Though every environment is different, increase your heap space until you at least have 2 GB available

The Java Heap Space is configured in $MW_HOME/user_projects/domains/soa_domain/bin/setSOADomainEnv.sh as follows:
PORT_MEM_ARGS="-Xms6144m -Xmx6144m ...
3. Set your PermSize (for Sun JDK only)

I recommend a min and max of either 1GB or 1.5GB for your PermSize.

The PermSize is configured in $MW_HOME/user_projects/domains/soa_domain/bin/setSOADomainEnv.sh as follows:
PORT_MEM_ARGS="... -XX:PermSize=1024m -XX:MaxPermSize=1024m ...

4. Set your Nursery Size (for JRockit only)

Generally speaking, the Nursery Size should be around 30% of your Java Heap Space.

The Nursery Size is configured in $MW_HOME/user_projects/domains/soa_domain/bin/setSOADomainEnv.sh as follows:
PORT_MEM_ARGS="... -Xns2048m ...

5. Use parallel garbage collection (for Sun JDK only)

Garbage collection (and some other recommended settings) is configured in $MW_HOME/user_projects/domains/soa_domain/bin/setSOADomainEnv.sh as follows:
PORT_MEM_ARGS="... -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:-TraceClassUnloading ...

6. Use JRockit instead of Sun JDK

My performance testing has shown an average of 32% performance improvement when using JRockit versus Sun JDK.

7. Modify and tune the Audit Level

Adjusting the Audit Level and tuning it is necessary.

1. Log in to the Fusion Middleware Console (at http://soaserver:7001/em)
2. Navigate to Farm_soa_domain --> SOA --> (right-click on) soa-infra --> SOA Administration --> Common Properties --> More SOA Infra Advanced Configuration Properties...
3. Click on "Audit Config"
4. Set the following values and click on "Apply" afterwards
AuditConfig/compositeInstanceStateEnabled = false    <-- you must set this for it to work!

AuditConfig/level = Production
AuditConfig/policies/Element_0/isActive = false
AuditConfig/policies/Element_0/name = Immediate
AuditConfig/policies/Element_1/isActive = true
AuditConfig/policies/Element_1/name = Deferred
AuditConfig/policies/Element_1/properties/Element_0/name = maxThreads
AuditConfig/policies/Element_1/properties/Element_0/value = 10
AuditConfig/policies/Element_1/properties/Element_1/name = flushDelay
AuditConfig/policies/Element_1/properties/Element_1/value = 5000
AuditConfig/policies/Element_1/properties/Element_2/name = batchSize
AuditConfig/policies/Element_1/properties/Element_3/value = 100
My performance tests have shown that changing from Development to Production results in an average of 46% performance improvement (without tuning the audit settings).

My performance tests have shown that changing from Production to Off results in an average of  61% performance improvement (without tuning the audit settings).

My performance tests have shown that if you apply the audit tuning described above, both Production and Off perform the same, effectively translating to a 61% performance improvement of the Production audit level.

8. Modify Mediator Service Engine properties (for Mediator only)

Manipulating the threads will have a direct impact on the ability of the server to process asynchronous Mediator services.

1. Log in to the Fusion Middleware Console (at http://soaserver:7001/em)
2. Navigate to Farm_soa_domain --> SOA --> (right-click on) soa-infra --> SOA Administration --> Mediator Properties
3. Edit the settings as follows:
Metrics Level:                                        Disabled
Parallel Worker Threads:                        12              <-- experiment increasing this value
Parallel Maximum Rows Retrieved:         600            <-- set to 50x the above setting
Parallel Locker Sleep Thread:                 1                <-- reduces waits for parallel routing rules
My performance tests have shown that changing from 4 worker threads and  200 max rows to 12 worker threads and 600 rows results in an average of 30% performance improvement for asynchronous Mediator services and 0% performance improvement for synchronous Mediator services.

9. Reduce soa-infra Log Levels

Unless required, reduce the SOA Suite Log Levels to error.

1. Log in to the Fusion Middleware Console (at http://soaserver:7001/em)
2. Navigate to Farm_soa_domain --> SOA --> (right-click on) soa-infra --> Logs --> Log Configuration
3. Set all Log Levels to ERROR:1 (SEVERE)
4. This must be repeated for all soa_server managed servers in your cluster

My performance tests have shown that reducing this from NOTIFICATION:1 (INFO) to ERROR:1 (SEVERE) results in an average of  7% performance improvement.

10. Perform Database Tuning

These are the database parameters I used in a large scale implementation. However, remember that they are based on the database AWR reports, so you may not want to just go ahead and apply them blindly. They are here for your reference and have resulted in some improvement in our environment.
AUDIT_TRAIL            = NONE
DB_FILES               = 1024
DISK_ASYNCH_IO         = TRUE
FILESYSTEMIO_OPTIONS   = SETALL
JOB_QUEUE_PROCESSES    = 10
LOG_BUFFER             = 209715200
NLS_SORT               = BINARY
SESSION_CACHED_CURSORS = 500
PROCESSES              = 1500
SESSION_MAX_OPEN_FILES = 50
UNDO_MANAGEMENT        = AUTO
PLSQL_CODE_TYPE        = NATIVE
MEMORY_TARGET          = 0
SGA_TARGET             = 6g
PGA_AGGREGATE_TARGET   = 2g
TRACE_ENABLED          = FALSE

11. Modify BPEL Process Manager Properties (for BPEL only)

Manipulating the threads will have a direct impact on the ability of the server to process asynchronous BPEL processes.

1. Log in to the Fusion Middleware Console (at http://soaserver:7001/em)
2. Navigate to Farm_soa_domain --> SOA --> (right-click on) soa-infra --> SOA Administration --> BPEL Properties
3. Edit the settings as follows:
Dispatcher System Threads:       10            <-- increase at least from the default value of 2 to 10
Dispatcher Invoke Threads:         20            <-- depends if your targets can handle the load
Dispatcher Engine Threads:        30            <-- should not exceed the sum of the two values above
Payload Validation:                    Disabled
Honestly, these numbers will vary based on the types of process designs you have.


Applicable Versions:
  • Oracle SOA Suite 11g (11.1.1.4)

14 comments:

Soumya Prakash Mishra said...

Hello Ahamad,

Great job done by you. I work as a SOA Admin and wanted to know if there is any particular platform recommended by Oracle for Fusion Middleware products like SOA and ADF(11g). We have many SOA/ADF 11g environments running on Sun T5240 machines+ Solaris 10. I feel the applications run a bit slow on them. For example the soa managed server to come up takes 25 minutes. Also I am using Sun JDK as recommended by Oracle instead of JRockit(for Solaris env) Any thoughts?

Ahmed Aboulnaga said...

Hey Soumya,

Here are a couple of comments to address your various questions.

1. Should you run SOA Suite on Solaris or Linux?

Oracle's position (you'll get the same response from HP as well) is that there is no preference on whether you want to run their products on Unix or Linux. Unfortunately, this is standard corporate speak and really doesn't help organizations in their decision making. Larry Ellison is quoted as saying, "I'm a Linux fan and if you want Linux we have the best Linux in the world. If you want Unix, we have the best Unix in the world."

In my opinion, Linux (Red Hat or OEL) is the preferred platform for Oracle SOA products, primarily for 3 reasons:

* PATCHING: Oracle patches are released on Linux before any other operating system.

* SKILLSETS: Linux dominates 90% of the market share versus 5% by Unix (including AIX, HP, Solaris).

* LICENSING COST: Obviously Linux is better.

2. Is SOA Suite 11g supposed to take a long time to start up?

Unfortunately, it does. Even in a new install with absolutely no composites deployed, it will take anywhere from 2-4 minutes to start up. Add another 1 minute for 10 composites deployed on a 2-node cluster. So if you have 200 composites, I wouldn't be surprised to see the startup time hovering around 25 minutes.

3. Should you use JRockit or Sun JDK with SOA Suite 11g?

JRockit is recommended in Linux, primarily because it is better in performance (as I've shown above), and you have better monitoring capabilities using JRockit Mission Control (jrmc). As you, I have also heard that Sun JDK is preferred on Solaris, but I have no data confirming why this is so.

Raja Ramasamy said...

Hi Ahmed,

Great post. Certainly helped us a lot. i have one questions about the "Dispatcher Invoke Threads" property? Do you have a guideline on how to measure the adequate value required for this property. In BPEL 10g, this default value was at a domain level. whereas in 11g, this is at the soa-infra level. As we migrate from 10g to 11g, should I just consider setting this in 11g to sum of the number of threads set in each 10g domain.

(note: In our production we expect around 4000 BPEL calls per minute)

Anonymous said...

this
Dispatcher Engine Threads: 30 <-- should not exceed the sum of the two values above

is not true

Ales Plsek said...

Hi Ahmed,

Thank you for this great post.

When reading the steps 7-9 and 11, I was wondering if you really need to use FM console or if there is a way howto enable these settings using a script?

Thank you,
Ales

Anonymous said...

Ales,

Yes, these can all be scripted. I'll be posting something that describes how to do so soon.

Yatan said...

Great post.

surya said...

Its really helpful.. Thanks for the post @Ahmed Aboulnaga

Jessica said...

Excellent information on performance tunning concept.. Thanks expecting many more information...

Nicky Paul said...

This wordpress module is made for shippers who need to offer items from their 3dcart Online Store inside their WordPress blog .
wordpress shop

Unknown said...

This is a great inspiring article. I am pretty much pleased with your good work. You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.Android Application Software

Unknown said...

Helpful Information
Freshers jobs way provide freshers jobs ,walkins,direct recruitment,openings for freshers and experienced and recruiting experienced and freshers Walkins

LENIN said...

The young boys ended up stimulated to read through them and now have unquestionably been having fun with these things. JAVA Training in chennai

Rekha said...

It is so good to read a new article.
java interview questions and answers
selenium interview questions and answers
digital marketing interview questions and answers
hadoop interview questions and answers
oracle interview questions
data science interview questions and answers