Back to the future

Back in 1993 I started my career at Oracle as a Unix System Administrator with a specialization in Sun Solaris. I have been using the Solaris operating system ever since the Solaris 2.0 Early Adopter program started in 1991. Those were the days ;-) In late 1998 I switched from being a system administrator to […]

Dtrace probes in Oracle 12c… v$kernel_io_outlier is populated by dtrace!!

Oracle 12c certainly has some great features, but for the performance guy like myself, performance monitoring features are particularly interesting.  There are three new v$ tables that track anomalies in the IO path.  The idea is to provide more information about really poorly performing IO that lasts more than 500ms.

  • V$IO_OUTLIER : tracks the attributies of an IO.  The size, latency as well as ASM information is recorded.
  • V$LGWRIO_OUTLIER : tracks information specifically on Log writer IO.

These two tables are going to be useful to monitor when performance issues occur.  I can already see the SQL scripts to monitor this activity starting to pile up.  But, there is one little extra table that dives even further into the IO stack using Dtrace.

Analyzing IO at the Exadata Cell level… a simple tool for IOPS.

Lately I have been drawn into to a fare number of discussions about IO characteristics while helping customers run benchmarks.  I have been working with a mix of developers, DBAs, sysadmin, and storage admins.  As I have learned, every group has there own perspective – certainly when it comes to IO and performance.

  • Most DBA’s want to see data from the DB point of view so AWR’s or EM works just fine.
  • Most System Admin’s look at storage from the Filesystem or ASM disk level.
  • Storage Admins want to see what is going on within the array.
  • Performance geeks like myself, like to see all up and down the stack <br />

    	  	<div class=

Solaris Eye for the Linux Guy… Part III (hugepages = ISM)

This post has been a long time coming but recently, I have started working on some SPARC SuperCluster POC’s with customers and I am getting re-acquainted with my old friend Solaris and SPARC.

If you are a Linux performance guy you have likely heard of HugePages.   Huge pages are used to increase the performance of large memory machines but requiring fewer TLB‘s .  I am not going to go into the details TLB’s, but every modern chip supports multiple memory page sizes.

So how do you get huge pages with Solaris?

Do nothing – it is the DEFAULT with Oracle running on Solaris.

Tuning is in the eye of the beholder… Memory is memory right?

It is human nature to draw from experiences to make sense of our surroundings.  This holds true in life and performance tuning.   A veteran systems administrator will typically tune a system different from an Oracle DBA.  This is fine, but often what is obvious to one, is not to the other.  It is sometimes necessary to take a step back to tune from another perspective.

I recently have ran across a few cases where a customer was tuning “Sorts” in the database by adding memory. Regardless of your prospective, every one knows memory is faster than disk; and the goal of any good tuner is to use as much in memory as possible.   So, when it was noticed by the systems administrator that the “TEMP” disks for Oracle were doing a tremendous amount of IO,  the answer was obvious right?

VirtualBox and Solaris 10 Guest Additions

Just a quick post really to save you some time-as it stands the guest additions for Solaris 10 are broken with Virtual Box 4.1.0. When I tried to install them in my Solaris 10 U9 guest, everything worked until the vboxsf module should have been loaded. The output from dmsg is as follows:

Aug 30 18:45:33 unknown genunix: [ID 819705 kern.notice] /usr/kernel/fs/amd64/vboxfs: undefined symbol
Aug 30 18:45:33 unknown genunix: [ID 819705 kern.notice] /usr/kernel/fs/amd64/vboxfs: undefined symbol
Aug 30 18:45:33 unknown genunix: [ID 819705 kern.notice] /usr/kernel/fs/amd64/vboxfs: undefined symbol
Aug 30 18:45:33 unknown genunix: [ID 472681 kern.notice] WARNING: mod_load: cannot load module 'vboxfs'

I’m running Virtual Box 4.1 on Windows 7. When I tried to check for updates the application returned “you are already running the current version”. However, checking the Downloads section of the website I found 4.1.2-which I’m trying now.

The workaround for 4.1.0 is to install older guest additions, see for more detail.

Linux takes a page from Solaris… pmap available on Linux.

Recently, there was a thread on an internal alias of old Sun guys.  The problem at hand was to track down a process that is consuming memory on Linux.  This is the type of problem that can be solved many ways (ps, top, etc…), but to my amazement someone mentioned that pmap could be used for Linux…. I guess I didn’t get the memo:)

About 6 months back I wrote a few entries that discussed Solaris tools for the Linux Guy in the posts:

Compiling iozone for Solaris 10 on SPARC

I have started working with ZFS and its various ways of protecting disks from failure. It’s a low end setup at best, where a JBOD is used as an extension to a M-series server. Many JBODs come with SAS connectivity only, and no on-board intelligence so a host based solution has to be chosen to protect the lot from disk failures.

For Solaris, you can use ZFS amongst other solutions. Alternatively, ASM is a possibility. I couldn’t reproduce the exact setup, so I had to improvise. Still I wanted to find out how ZFS performs compared to ASM. For this purpose I used an EMC VMX and a Sun 5410 server which had 10 multipathed 10G LUNs presented to it via EMC Power Path 5.3.

To test the file system performance I decided to use both IOZone and Bonnie++. Bonnie++ is no problem, you can get it from Sun Freeware. Note that Oracle no longer produce the Companion CD, which leaves SunFreeware the only source for alternative packages.

Before you can compile anything on Solaris, you need a compiler (why doesn’t Solaris come with one?). Installing the compiler was simple. I got the required files from these URLS:

They can be installed by unzipping and pkgadd:

bash-3.00# pkgadd -d gcc-3.4.6-sol10-sparc-local

The following packages are available:
  1  SMCgcc     gcc
                (sparc) 3.4.6

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:

Processing package instance  from 

gcc(sparc) 3.4.6

The selected base directory  must exist before
installation is attempted.

Do you want this directory created now [y,n,?,q] y
Using  as the package base directory.
## Processing package information.
## Processing system information.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

Installing gcc as 

## Installing part 1 of 1.
[ verifying class  ]

Repeat the procedure for libiconv as well.

Once that is done, it’s time to compile iozone. Get the latest tar ball from (it was iozone3_397.tar in my case) and unzip it somewhere on your system. Change directory to stageDIR/src/current. For the makefile to work, set your path to include /usr/local/bin and /usr/ccs/bin. The makefile comes with different targets depending on your architecture and compiler. In my case I thought that Solaris10gcc-64 would be most appropriate. Trying this option however didn’t succeed:

bash-3.00# make Solaris10gcc-64

Building iozone for Solaris10gcc-64

gcc -O -c  -Dunix -DHAVE_ANSIC_C -DASYNC_IO -D__LP64__ \
                -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Dsolaris \
                 -m64 libbif.c -o libbif10-64.o
gcc -O -c  -Dunix -DHAVE_ANSIC_C -DASYNC_IO -D__LP64__ \
                -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Dsolaris \
                -DNAME='"Solaris10gcc-64"'  -m64 libasync.c -o libasync10-64.o
gcc -c -O -Dunix -DHAVE_ANSIC_C -DASYNC_IO \
                -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Dsolaris \
                -DNAME='"Solaris10gcc-64"'  -m64 iozone.c -o iozone_solaris10gcc-64.o

Building fileop for Solaris10gcc-64

gcc -c -O  -m64 fileop.c -o fileop_Solaris10gcc-64.o

Building the pit_server

cc -c   pit_server.c  -o pit_server.o
gcc  -O  -m64 iozone_solaris10gcc-64.o libasync10-64.o libbif10-64.o \
        -lthread -lpthread -lposix4 -lnsl -laio \
        -lsocket -o iozone
gcc  -O -m64 fileop_Solaris10gcc-64.o -o fileop
gcc  -O -m64 pit_server.o -lthread -lpthread -lposix4 \
        -lnsl -laio -lsocket -o pit_server
ld: fatal: file pit_server.o: wrong ELF class: ELFCLASS32
ld: fatal: File processing errors. No output written to pit_server
collect2: ld returned 1 exit status
*** Error code 1
make: Fatal error: Command failed for target `Solaris10gcc-64'

ELFCLASS32? I specified that I wanted 64bit! Looking at the output a bit closer it turned out that cc (which I sym-linked to gcc in /usr/local/bin) created a 32bit object file. This was easy to fix. Instead of

# cc -c   pit_server.c  -o pit_server.o

I executed

# gcc -m64  -c pit_server.c  -o pit_server.o

That created a 64bit object file:

bash-3.00# file pit_server.o
pit_server.o:   ELF 64-bit MSB relocatable SPARCV9 Version 1

Now the linking worked as well:

bash-3.00# gcc  -O -m64 pit_server.o -lthread -lpthread -lposix4 \
>         -lnsl -laio -lsocket -o pit_server
bash-3.00# echo $?

And finally pit_server was 64 bit:

bash-3.00# file pit_server
pit_server:     ELF 64-bit MSB executable SPARCV9 Version 1,
                dynamically linked, not stripped, no debugging information available

Happy benchmarking

Automatic log gathering for Grid Control 11.1

Still debugging the OMS problem (it occasionally hangs and has to be restarted) I wrote a small shell script to help me gather all required logs for Oracle support. These are the logs I need for the SR, Niall Litchfield has written a recent blog post about other useful log locations.

The script is basic, and can possibly be extended. However it saved me a lot of time getting all the required information to one place from where I could take it and attach it to the service request. Before uploading I usually zip all files into to avoid clashing with logs already uploaded. I run the script via cron daily at 09:30.


# A script to gather diagnostic information for OMS GC 11.1 for use by Oracle
# Support
# WARNING the previous output of the script's execution is NOT preserved!
# (c) Martin Bach Consulting Ltd All rights reserved
# The script performs the following tasks
# 1) creates a heap dump for the EMGC_OMS1 server
# 2) creates a compressed tar archive of all server logs
#    including the webtier

set -x


# get heap dump for EMGC_OMS1-I upped the JVM args to -Xms of 768M which works
# fine as an identifier in Solaris 10. For Linux you may have to grep for EMGC_OMS1
# in the ps output
# the heap dump goes into EMGC_OMS1.out which is compressed and saved later
DUMP=`ps -ef | grep "Xms768m" | grep -v grep | awk '{print "kill -3 "$2}'`

if [ ! -d $DST_DIR ]; then
echo $DST_DIR does not exist-creating it
mkdir -p $DST_DIR

# get the relevant logs
cd $GC_INST/em/EMGC_OMS1/sysman/log/
tar -cvf - . | gzip > $DST_DIR/gc_inst.em.EMGC_OMS1.sysman.log.tar.gz

cd $GC_INST/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/
tar -cvf - . | gzip > $DST_DIR/

cd $GC_INST/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/
tar -cvf - . | gzip > $DST_DIR/

cd $GC_INST/WebTierIH1/diagnostics/logs/OHS/ohs1
tar -cvf - . | gzip > $DST_DIR/webtier.ohs1.log.tar.gz

cd $GC_INST/WebTierIH1/diagnostics/logs/OPMN/opmn
tar -cvf - . | gzip > $DST_DIR/webtier.opmn.log.tar.gz

Good luck!

Solaris Eye for the Linux Guy… Part II (oprofile, Dtrace, and Oracle Event Trace)

Proper tool for the job

My grandfather used to say to me: “Use the proper tool for the job”.  This is important to keep in mind when faced with performance issues.  When I am faced with performance problems in Oracle, I typically start at a high level with AWR reports or Enterprise Manager to get a high level understanding of the workload.   To drill down further, the next step is to use Oracle “10046 event” tracing.  Cary Millsap created a methodology around event tracing called “Method-R” which shows how to focus in on the source of a performance problem by analyzing the components that contribute to response time.   These are all fine places to start to analyze performance problems from the “user” or “application” point of view.  But what happens if the OS is in peril?