Search

Top 60 Oracle Blogs

Recent comments

Oakies Blog Aggregator

Little Things Doth Crabby Make – Part XIII. When Startup Means Shutdown And Stay That Way.

This seemed worthy of Little Things Doth Crabby Make status mostly because I was surprised to see that Oracle Database 11g STARTUP command worked this way… Consider the following text box. I purposefully tried to do a STARTUP FORCE specifying a PFILE that doesn’t exist. Well, it did exactly what I told it to do, [...]

I should've been in movies...


Have you seen the movie Avatar? If not, this will just look like a crazily distorted blue photo of me. But, if you saw the movie, you'll see me as a Na'vi. Cool, huh?!?

I loved the movie (it was fabulous in 3D!) and if you haven't seen it yet, check out the trailer.

I wish (2)

Continuing my little list of things in SQL Server that I’d like to see in Oracle: 1 Index statistics: when SQL Server collects index statistics, it also collects stats not only on the full index definition, but also on every “prefix” of the index, i.e. if you have a 4-column index, you will get stats [...]

Instrumentation rocks!!

Those who have read my previous posts know I've got a few favorite topics:1. Data Models2. Measuring3. Variance4. InstrumentationThis post will be short but it touches on all of my favorite things. Over the last year and a half, I've had the opportunity to redesign a not-so-great application working with the amazingly great developers at Method-R. In the process, we redesigned the data model,

Oracle Mix – OOW 2010 Suggest-A-Session

I hope it’s not too late. I just finished my Suggest-A-Session on Oracle Mix titled “Mining the Oracle Database AWR repository for Capacity Planning“, this might be of interest to you… below is the abstract:

“New CPUs and storage arrays, they are getting faster but these resources are finite and come at a cost. Hence, capacity planning plays a very important role to ensure proper resources are available to handle expected and unexpected workloads. Another critical matter for the DBAs and IT managers is justifying the expense of adding resources on the system. With guesswork you’ll end up getting the most expensive hardware. With proper measurement, proper planning and management of growth, you’ll be able to get just the right hardware for your workload with allowance for a particular growth period. This will result to huge savings for the company and a happier IT shop.

AWR is a built-in data store that started in 10gR1 and it’s very much like a “Statspack on steroids”. It has improved significantly in 11gR2. What’s exciting about 10g and higher versions is we could utilize the “Time Model” and the “OSSTAT” view and use them together with the “SYSTAT” view to have a far better workload information when going through all the AWR snapshots.

From the AWR data samples, we could build amazing reports that will let us notice trends and makes it possible to visualize data and use statistical methods for analysis. Even more surprising about the AWR data samples is we are able to define the database server’s Capacity, Requirements, and Utilization in terms of CPU, IO, memory, and network which are very important key metrics for Capacity Planning.

In this session, attendees will be able to learn how to make use of the AWR specifically the DBA_HIST tables to have a clear-cut measurement on resources to aid in Capacity Planning, Predictive Analysis, and Performance Firefighting.”

it is really cool I’m telling you… c’mon, just click on the VOTE button </p />
</p></div>

    	  	<div class=

Running Oracle Database On A System With 40% Kernel Mode Overhead? Are You “Normal?”

Fellow Oak Table Network member Charles Hooper has undertaken a critical reading of a recently published book on the topic of Oracle performance. Some folks have misconstrued his coverage as just being hyper-critical, but as Charles points out his motive is just to bring the content alive. It has been an interesting series of blog [...]

How to kill performance with 16MB table

If you think 16MB tables are so small that can’t be root cause of significant performance problems with SELECT queries, then read about my today’s headache – a story of 16MB table worth something like 270 seconds of response time. It began mostly as usual: an application web page didn’t show up within 5 minutes. [...]

Little Things Doth Crabby Make – Part XII.1 Please, DD, Lose My Data! I Didn’t Need That Other 4K Anyway.

I’ve never done a “dot release”  for any post in my Little Things Doth Crabby Make series but there is a first time for everything. In yesterday’s post (Part I) I blogged about how dd(1) on Linux will happily let you specify arbitrarily huge values for the bs (block size) option yet doing so will [...]

New Public Oracle Security Training Class Dates announced

I have just agreed four new public Oracle Security classes to be taught this year. All of the new classes are our very popular two day class " How to perform a security audit of an Oracle database ". These....[Read More]

Posted by Pete On 17/06/10 At 03:57 PM

Configuring Large PeopleSoft Application Servers

Occasionally, I see very large PeopleSoft systems running on large proprietary Unix servers with many CPUs.  In an extreme case, I needed to configure application server domains with up to 14 PSAPPSRV processes per domain (each domain was on a virtual server with 8 CPU cores, co-resident with the Process Scheduler).

The first and most important point to make is don't have too many server processes.  If you run out of CPU or if you fully utilise all the physical memory and start to page memory from disk, then you have too many server processes.  It is better to queue on a Tuxedo queue rather than the CPU run queue, or disk queue during paging.

Multiple APPQ/PSAPPSRV Queues

A piece of advice that I originally got from BEA (prior to their acquisition by Oracle) was that you should not have more than 10 server processes on a single queue in Tuxedo.  Otherwise, you are likely to suffer from contention on the IPC queue structure because processes must acquire exclusive access to the queue in order to enqueue a service request to the queue or dequeue a request from it.  Instead multiple queues should be configured that are both serviced by the same server processes and so advertise the same services. 

If you look at the 'large' template delivered by PeopleSoft, you will see that it produces a domain that runs between 9 and 15 PSAPPSRV processes.  This does not conform to the advice I received from BEA.  I repeated this advice in PeopleSoft for the Oracle DBA.  Though I cannot now find the source for it, I stand by it.  I have recently been able to conduct some analysis to confirm it on a real production system.  Domains with two queues of 8 PSAPPPSRV server process each out performed domains with only a single queue.

Load Balancing Across Queues

If the same service is advertised on multiple queues, then Tuxedo recommended that you should specify realistic service loads and use Tuxedo load balancing to determine where to enqueue requests.  I want to emphasise that I am talking about load balancing across queues within a Tuxedo domain, and not about load balancing across Tuxedo domains in the web server.

This is what the Tuxedo documentation says about load balancing:

"Load balancing is a technique used by the BEA Tuxedo system for distributing service requests evenly among servers that offer the same service. Load balancing avoids overburdening some servers while leaving others idle or infrequently used. Before sending a request to a service routine, the BEA Tuxedo system identifies all servers capable of handling the request and selects the one most appropriate for maintaining a balanced load across all the servers in the configuration.


You can control whether a load-balancing algorithm is used on the system as a whole. Such as algorithm should be used only when necessary, that is, only when a service is offered by servers that use more than one queue. Services offered by only one server, or by multiple servers in a Multiple Server, Single Queue (MSSQ) do not need load balancing. The LDBAL parameter for these services should be set to N. In other cases, you may want to set LDBAL to Y."

It doesn't state that load balancing is mandatory for multi-queue domains, and only hints that it might improve performance.  If load balancing is not used, the listener process puts the messages on the first empty queue (one where no requests are queued).  If all queues have requests the listener round-robins between the queues.

You could consider giving ICScript, GetCertificate and other services with small service times a higher Tuxedo Service priority.  This means they jump the queue 9 times out of 10.  ICScript is generally used during navigation, GetCertificate is used at log on.  Giving these services higher priority will mean they perform well even when the system is busy.  Users often need to do several mouse clicks to navigate around the system, but these services are usually quick.  This will improve the user experience without changing the overall performance of the system.

Data

I have recently been able to test the performance of a domains with up to 14 PSAPPSRVs on a single IPC queue, versus domains with two queues with up to 7 PSAPPSRVs each, both with and without Tuxedo queue balancing.   These results come from a real production system where the multiple queue configuration was implemented on 2 of the 4 application servers.  The system has a short-lived weekly peak period of on-line processing.  During that time Tuxedo spawns additional PSAPPSRV processes, and so I get different sets of times for different numbers of process. 

The timings are produced from transactions sampled by PeopleSoft Performance Monitor.  I capture the number of spawned processes using the Tuxmon scripts on my website that use tmadmin to collect Tuxedo metrics.

1 Queue 2 Queue
Server Processes per Queue Number of Services Mean ICPanel Service Time Server Processes per Queue Number of Services Mean ICPanel Service Time
6 2,616 1.33 3 6945 1.05
7 1,949 0.97
8 1,774 1.06 4 7595 1.16
9 1,713 1.02
10 1,553 1.25 5 4629 1.17
11 1,250 1.30
12 969 1.32 6 3397 1.16
13 427 1.21
14 1,057 1.10 7 3445 1.13
 Total
13,308 1.17 26011 1.13

The first thing to acknowledge is that this data is quite noisy because it comes from a real production system, and the effects we are looking for are quite small.

I am satisfied that the domains with two PSAPPSRV queues generally perform better under high load, than those under 1.  Not only does the queue time increase on the single queue domain, the service time also increases.

However, I cannot demonstrate that Tuxedo Load Balancing makes a significant difference in either direction.

My results suggest that domains with multiple queues for requests handled by PSAPPSRV process perform slightly better without load balancing if there is no queue of requests, but perform slightly better if there is a queue of pending requests.  However, the difference is small.  It is not large enough to be statistically significant in my test data.

Conclusion

If you have a busy system with lots of on-line users, and sufficient hardware to resource it, then you might reach a point when you need more than 10 PSAPPSRVs.  In which case, I recommend that you configure multiple Tuxedo queues.

On the whole, I would recommend that Tuxedo Load Balancing should be configured.  I would not expect it to improve performance, but it will not degrade it either.