Tuning Oracle Traffic Director (OTD) For SOA, OSB, PeopleSoft, EBS, Webcenter Portal

One of the problem which I’ve met by one of our customer is SOA performance problem.

The environment by the customer has webcenter portal, SOA, OSB, Oracle Unififed Directoery.

Alle these environments are running on Exalogic/exadata with OTD (oracle Traffic Director) configured for load balance.

On a regular time we found out that SOA server generate the following errors

-          Crashed with out of Memory error

                “””Error>              

                by the underlying routing system. Root cause :java.lang.OutOfMemoryError:   

                allocLargeObjectOrArray”””

-           Response time of the SOA server is very slow

-           No response at all

-          Enable to invoke endpoint URI  “XXXXXXXXXXX:8002/services/” successfully due to: java.xml.soap.SOAPException: Bad response:504 Gateway Timeout

-          A lot of StuckThreads on the SOA servers

This is an OTD problem. Most of SOA suite requests are POST requests in other words every request between composites will generate NEW HTTP connection at the remote SOA server. This will cost a lot of overhead connection.hich are slow to garbage collect for technical reasons. Eventually, with sufficient load, the pile up will be so great that SOA ends up stuck in a Garbage collection loop and will either throw OutOfMemoryError or slow response.

The key to solve this problem is on the OTD side. There is a paramter in the OTD which will help dramatically to solve this problem & improve dramatically the SOA’s performance.

“ Always use keep-alive ”, By default this parameter is not enable . By enable this parameter OTD will cache and reuse the same HTTP connection for multiple requests (it actually has a small pool of cached connections. This Parameter needs to add to Origin Servers.

Beside  this parameter there are some other parameters which are helpful to increase performance on the SOA environment. I’ll mention to some of them with their function.

·         Enable keep-alive: This parameter indicates whether the Oracle Traffic Director virtual server should attempt to use persistent connections to the origin server or create a new connection for each request. It is enabled by default.

·         Keep-alive timeout: This parameter specifies the maximum duration, in seconds, for which a persistent connection can be kept open. The default timeout duration is 29 seconds.

·         Idle timeout: This parameter specifies the maximum duration, in seconds, for which a connection to the origin server can remain idle. The default duration is 300 seconds.

·         Always use keep-alive: This parameter indicates whether the Oracle Traffic Director virtual server can reuse existing persistent connections to origin servers for all types of requests. If this parameter is not enabled (default), the Oracle Traffic Director virtual server attempts to use persistent connections to the origin server only for the GET, HEAD, and OPTIONS request methods.

·         Proxy buffer size: This parameter specifies the size of the buffer in which Oracle Traffic Director stores data received from the origin server, before sending the data to the client. Larger the buffer, lower is the number of write system calls. The default size of the proxy buffer is 16 kilobytes.

Steps to implement this change

Changes on the OTD can be executed through administration console or CLI (command-line)

To user CLI, we  need to perform the following steps

-          add SOA-obj.conf file this file is located under INSTANCE_HOME/net-config_name/config

In my example I added the following lines:

-          save the file

-          Run pull-config to update configuration store on Admin server & to push new configuration on all instances

-          tadm> pull-config --config=config_name node 

A simple way to achieve this is by using Administration Console 

 - login to Oracle Traffic Director Console 

           - Configuration-> select SOA-> Virtual Servers-> Routes-> default-route

 

     

     

-        - Advanced Settings under tab Configuration for Connections with Origin Servers Change the following settings:

 

1.       enable Always use Keep Alive (default disable)

2.       Increase Buffer Size to 32768, default ( 16KB)

3.       Idle Timeout 360, default (300 seconds)

-         - Save

-          - Deploy Changes

This will prevent the overhead from creating a TCP/IP connection for every request hence improve performance dramatically. And also, since there will be a smaller number of the connections to the origin server (for example SOA, OSB,PeopleSoft, EBS & WebCenter Portal, etc. ), it will reduce the chance of build up of stale connections on the remote server.

For more details please check this URL http://docs.oracle.com/cd/E23389_01/doc.11116/e21036/perf014.htm