#555555;">Note:

#555555; padding-left: 30px;">This is a reposting of an old blog post that was on dboptimizer.com but is no longer accessible

#555555; padding-left: 30px;">More trace file analyzer tools at  #2970a6;" href="http://ba6.us/node/177">http://ba6.us/node/177

#555555; padding-left: 30px;">Related blog post: Oracle “Physical I/O” ? not always physical with a 10046 parser specifically for I/O parsetrc.pl and readme

#555555;">Often when I have a 10046 trace file, especially when looking at I/O issues, I want a histogram of I/O response time.  To get I/O response time I’ve hacked out incomple awk scripts from time to time, always meaning to write a more complete one, well now I don’t have to. It’s already been done!

#555555;">Here is a cool perl script from Clive Bostock: #2970a6;" href="https://sites.google.com/site/oraclemonitor/10046-pl-readme">README.TXT    #2970a6;" href="https://sites.google.com/site/oraclemonitor/10046-pl-1">10046.pl

#555555;">(also checkout #2970a6;" href="http://www.oracledba.ru/orasrp/">orasrp which produces a more indepth report in HTML. I like both. I like 10046.pl as a short  easy portable script that I can modify, whereas orasrp is a binary and only works on some ports)

#555555;">For example, if I trace a session with 10046, and retrieve the tracefile, then I can run:

#555555;">$ 10046.pl -t  mytrace.trc

#555555;">and it will output  a header and three sections

#555555;">Header

#555555;">
• Summary of all events for tracefile
• Events by object summary
• Events by object histogram

#555555;">This looks like

#555555;">Header

#555555;">Trace file mytrace.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u02/oracle
System name:    SunOS
Node name:      toto
Release:        5.10
Version:        Generic_142900-12
Machine:        sun4u
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 177
Unix process pid: 16553, image: oracle@toto
Trace input file : mytrace.trc

#555555;">Wait summary

#555555;">EVENT AGGREGATES
================
Wait Event              Count Elapsed(ms)   Avg Ela (ms)  %Total
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~ ~~~~~~~~~~~~ ~~~~~~~~~~~~ ~~~~~~~~~~
db file sequential read  2715      11593              4    3.74
       direct path read  4484       4506              1    1.45
 db file scattered read   141        898              6    0.29
          log file sync     3          8              2    0.00
                              ~~~~~~~~~~~~
             Total Elapsed:       309821

#555555;">Wait Summary by object

#555555;">Object Id  : Wait Event                  Count Tot Ela (ms) %Total Avg Ela (ms)
~~~~~~~~~~ : ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~ ~~~~~~~~~~~~ ~~~~~ ~~~~~~~~~~~~~
28581      : direct path read            4484         4506   1.45            1
1756433    : db file sequential read      725         1891   0.61            2
764699     : db file sequential read      332         1762   0.57            5
37840      : db file sequential read      200         1044   0.34            5
38018      : db file sequential read      108         1009   0.33            9
81596      : db file scattered read       140          887   0.29            6

#555555;">wait histogram by object

#555555;">EVENT HISTOGRAM BREAKDOWN
===========================
This section splits the event counts into elapsed time
buckets so that we can see if there are any suspiciousn
or anomalous response time / frequency patterns.
Object Id : Wait Event              <1ms <2ms <4ms <8ms <16ms <32ms <64ms <128ms <256ms <512ms >=1024ms
~~~~~~~~~ : ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~~ ~~~~~~ ~~~~~~ ~~~~~~~~
28581     : direct path read        7680   87  148  221  144    40     4     0      0       0        0
1756433   : db file sequential read  606  268   45   35   66     6     2     0      0       0        0
764699    : db file sequential read   74  119   11   78   78     9     0     0      0       0        0
37840     : db file sequential read   50   72    6   45   47     5     0     0      0       0        0
38018     : db file sequential read   12   38    7   10   30    12     5     0      0       0        0
81596     : db file scattered read    64    4   13   62   18     8     3     0      0       0        0
41995     : db file sequential read   20   39    0    7   16     8     4     0      0       0        0
108718    : db file sequential read   74   54    5   12   24     4     0     0      0       0        0
33490     : db file sequential read    0    5   11   25   19     4     0     0      0       0        0

The term “Service” is used to represent numerous objects within Enterprise Manager 12c.  Although the most common reference to the term is related to the Oracle Management Service, (OMS), in this blog post, we are going to drill down and refer to all services identified in the drop down menu from Targets, Management Services and Repository.

The two main services we are going to cover here are set up as part of the Oracle Enterprise Manager 12.1.0.4 installation and shouldn’t be edited or reconfigured, but I think it’s good to know where everything is if you should ever need to investigate it or are working with Oracle Support.

By default the following services will display in the EM12c environment-

• EM Console Service
• EM Job Service

These two services are essential to successful Enterprise Manager task completion.  The EM Console Service is the service that keeps the console active and accessible and the EM Job Service controls not just the EM Jobs viewable in the EM Job Activity console, but also a ton of background processing for the Enterprise Manager.

The Main Console View

Once you’ve accessed the services from the target menu, as these are both monitored targets in the Enterprise Manager, you’ll see the following information:

 

Along with the two, default services we expect to see for a standard Enterprise Manager installation, we also can see the status, if there are any open incidents and what system the services belong to.  You can create and remove services from here, but unless you know what you’re doing, this is mostly the view to quickly filter service information.

Creating services from this console offers you three, two generic service types and one advanced service option called an Aggregate service.

Using our existing services as examples, it may give you a bit of an idea of how a System Based service is different than a Test Based one.

A System based service monitors one single system vs. a Test based one, which monitors a one or more test based systems.  Notice that the EM Jobs Service does ONE thing-  it monitors the Enterprise Manager Job Service.  The EM Console Service has to monitor EVERYTHING that is involved in the console, which is much more complex and requires a constant test to ensure all aspects are available.

And Aggregate Service is a combination of sub-services.  If one of the many sub-services is experiencing an issue, the entire Aggregate service is dependent upon that one failure.

There are a number of other services that Enterprise Manager 12c supports:

• Business Application
• Service Entry Point
• Oracle Database Service
• Oracle Applications Service
• Siebel Services
• Beehive Services
• EM Service
• And others….

You can create, manage and remove services via EM CLI commands, too.  The following commands will support the management of each:

• create_service
• assign_test_to_target
• change_service_system_assoc
• remove_service_test_assoc
• set_availability
• set_properties
• And many, many others….

To the right of the console list of services, are Key Components incident information.  This can assist the administrator when they are experiencing a problem, but the culprit isn’t displayed in the service, a related target may be the issue:

 

The Key Tests also show current status and any monitoring beacons that are attached to the service.

EM Console Service

If we click on the EM Console Service, we are taken to the home page for the service.

Centering in on the upper, left section of the home page, you notice that there are a number of tabs:

• Home
• Charts
• Test Performance
• System
• Monitoring Configuration
• Topology

For any generic service, you are going to see these tabs, but we’ll go through each one of the tabs, (outside of the home tab) for the EM Console Service so that you get a good feel for what is available in each and what tab has control over the configuration of the service.

Charts

Upon clicking the charts menu, you’ll see the basic information about page response per millisecond and page hits per minute:

Unless there is a serious performance issue in the console, I commonly don’t find a lot of useful data from the default view of “last 24 hrs” and recommend changing the view to [at least] the last 7 days.  You’ll be surprised how different the data will look when you take into account the nightly maintenance jobs that keep EM12c clean and effective:

Test Performance

The next tab is for test performance.  All services have tests involved that check and see if a service is actively running and passes either one or more tests that verify it is properly functioning and available.

The value for the last EM Console Service Test is shown above in the graph, but the real information lies lower in the performance timestamp.  The test isn’t run very often, (it is a performance impact to the system, so full testing of the service is only run once every 24 hours.)  You can see the Collection Timestamp in the right hand window of 6:55am, which will also match the extended Web Transaction used to verify that the console is accessible if you click on the EM Console Service Test at the bottom.

System

The System tab displays the latest information about the status and incidents of the components connected to the service.  

As you can see from the screen shot above, a recent test of a non-released patch has caused some grief, (this is why we test these things… :)) and there are a few applications components that were impacted and need to be reviewed and addressed.  Each of the components are linked on this page, so they can be easily accessed and the incident investigated.  As this is the EM Console Service, there are a number of related components, (16 total, as shown in the right hand totals) and you can change to the next page to display the rest of the components involved.

Monitoring Configuration

The Monitoring Configuration requires a section to itself, as this has some pretty impressive links in this tab.  We are going to go through each one of these, so you can get a solid understanding of what is available:

System Association

First thing you need to remember, DON’T CLICK ON THE “REMOVE SYSTEM” BUTTON.  Although I’ve never seen anyone attempt this from the console, I can’t imagine the damage that would be done if you chose to remove a key component from the EM Console Service unless instructed by Oracle Support, so just don’t attempt it to see what happens…

As discussed above, each component associated to the EM Console Service is displayed, along with the designation as a key component with a check mark.

Root Cause Analysis Configuration

By setting the analysis mode to manual, less stress is put on the system resources to collect root cause data at any issue, letting you control when the analysis is performed.  The recommendation is to leave this set to manual mode for analysis collections and only to change it [again] with the blessing of Oracle Support.

If any root cause analysis has been performed, you would view it to the right of the list of components and types, as the bottom section is reporting info only, listing each component and any test counts.

Service Tests and Beacons

This is how cloud control knows that the console service and key components are up and available.  A set of tests are run on a regular interval in the form of a web transaction.  This means that the test logs in, verifies a result from the URL for cloud control and verifies each key component is responsive.  If all tests result in a positive result, then the status is ‘up’.

A beacon is a target used to monitor service tests, primarily to measure performance of the service. One beacon belongs to at least each service test and is used to check connectivity between the client and the OMS host or the OMS host and the OMR host, etc.  Adding multiple beacons for one service adds no value, so if for some reason, you see multiple beacons assigned to a service, there’s a problem right there.

Availability Definition

This console looks very similar to the one above, but it covers test information about the EM Console Service Tests and EM Management Beacon:

By default, both of these require ALL KEY SERVICE TESTS must be SUCCESSFUL to be marked successful and return a status of ‘up’.  As you can see, the EM Management Beacon has not done so, which results in an unavailable status.  The next step in trouble-shooting this would be to click on the Beacon name link and investigate what  didn’t return successfully.

Performance Metrics

Performance metric data is where you set your thresholds for EM Console Service response, (and the readers say “Aha!  That’s another location I can get to the valuable metric threshold setting!”)

The metric value settings for the threshold are in milliseconds and if you are receiving notifications that this is beyond your response time, look at network connection, network connection between data centers, etc. and if it’s something you can’t correct, bump up the warning and critical threshold values to provide more value.

Usage Metrics

This setting by default isn’t set, but it’s one area, you can configure if you would like.  If you are concerned about page performance and want to report on this metric data, set up the with a logical value to start.  For a small web server, 12000 page hits per minute would be a pretty good warning value with 20000 page hits per minute critical.  Use this as a beginning base, test, rinse and repeat.

Now we’ve completed a high level review of the Monitoring Configuration, we can return to the last tab, Configuration Topology.

Topology

This last tab is for those of you that love topology charts.  This is the ultimate in configuration topology, allowing you to interact with all data, component dependency and connections to the EM Console Service.

 

This is also where you come to realize how dependent everything is on your EM Console Service-  no website, no Enterprise Manager access.  By hovering my cursor over the service, you can quickly see the dependent HA servers, database instance, management agent and other essential hardware and software components that are involved with the EM12c hardware and framework.  To the right is filtering options to drill down into a more manageable view:

High level information about each of the targets you want to drill down on are displayed and you can use this section to create a much easier to read topology:

 

The topology makes even more sense if you can view all the information on each of the targets vs. me hiding host/database names…

Hopefully this review of services helps explain a little bit more about this very, very important key component to Enterprise Manager 12c.  Next post on Services, I’ll go over the all important, EM Job Service.

 

Tags:  EM12c, Enterprise Manager, Services

Del.icio.us

Facebook

TweetThis

Digg

StumbleUpon

Photo by Kevin Dooley

If you are in the Bay Area tomorrow, Jan 27, come see myself, Yann Ropars and #141823;">Yury Velikanov talk about how to leverage social media to boost your career. We will be talking at Oracle head quarters at the NoCOUG/BIWA conference at #444444;">2:30 pm 

Why use social media as an IT technician? Because

vibrant social networks are key to landing jobs, moving forward in your career, and securing personal happiness

- Richard Florida #252525;"> professor #0b0080;" title="Rotman School of Management" href="http://en.wikipedia.org/wiki/Rotman_School_of_Management">Rotman School of Management#252525;"> at the #0b0080;" title="University of Toronto" href="http://en.wikipedia.org/wiki/University_of_Toronto">University of Toronto

#000000;"> Success no longer comes from possessing knowledge; instead, you have to participate with others in creating a flow of knowledge. 

- Walter Isaacson, President and CEO, the Aspen Institute, and author of Einstein: His Life and Universe

#444444;">Networks matter – who do you turn to for help? Who can you trust? How do you get a new job? How do you know what products to buy? What are the latest discoveries? How do you know if 12c is ready for prod?

Networks of people can accomplish things an order of magnitude faster than a lone individual. Look at wikipedia or open source software.

My first big social networks “ah ha” was 15 years ag  with a group of Oracle technicians  that Mogens Nørgaard had created an email list for. With the help from several different people on this list such as Anjo Kolk, James Morle and Jonathan Lewis, I was able to put together a program in a week to read Oracle’s SGA directly with C instead of SQL. Really cool stuff! Without each of their input I might have spent a year instead of a weekend.

That was 15 years ago and I’m convinced  more than ever that social networks are hugely important

My second big social network “ah ha” came in the world economic meltdown in 2009.  My company at the time wasn’t strong but I was happy there because I was running my own project and was head down working on my project ignoring the rest of the world. Then when the economy slipped I realized that my company was in a bad position as they cut salaries across the board, laid people off and cut my project team down to a handful and dumped 4 other software projects on by desk that I had no interest in. With almost no time to work on my project that I was passionate about, hardly any team left, more work than I could handle on other weak software packages and a lower salary, I was chomping at the bit to get out, but I realized I didn’t have any one one turn to.  All my contacts were at Oracle which I didn’t want to go back to and my current company. I realized I need a strong Linkedin network to leverage to find a new job. Ultimately it was the network that Mogens had set up that saved me and connected me to Delphix. Before this experience I had thought linkedin was just a place to put your resume online. I didn’t realize it was a strong social network that could actually help me get a job.

Now days, if I meet a DBA who is not on Linkedin, I assume that the person is a 9am-5pm “lifer” at the company they are currently at. It leads me to think they are happy where they are, have no plans on ever leaving, and no ambition to leave.  My assumptions are probably often wrong. Maybe the person just doesn’t know the power of social networks but there are so many people on social media who are easy to connect with, why bother with people who are hard to connect with?  In person connections were the only way to connect 20 years ago. In person connection, or local connections are still important, which is why Silicon Valley is so powerful. Silicon Valley is so powerful because there are so many smart, motivated, passionate IT technicians who can easily hop from one job to another, meet at bars, meetups, hackathons, conferences etc. But now days online social connections are becoming more and more important. When communication is rapid, things happen faster.

Social media for me is

• email groups, like Oracle-L
• forums, like OTN
• twitter
• linkedin
• facebook

5 ways to boost your career with social media:

1. Find jobs
   • become more well known by your presence
   • create a network of people you can turn to and who can validate your skills
   • share your knowledge and generate
2. Find employees
   • someone send you a resume? See who they are connected to. Are they connected to a strong network?
3. Solve technical problems
   • have an error, ask if others have seen it
4. Create projects together
   • github – get input and help on scripts and projects
5. Get feedback on technology
   • what’s new
   • what’s useful
   • what can you ignore
   • what problems should you watch out for

How do you get started?

Hop on twitter. Find someone important who only follows a few people. For example in the Oracle world, go to Tom Kyte‘s twitter profile.  Tom is only following 22 people and you can bet those 22 people are probably pretty good people to follow, so go through his list and pick out most or all and follow them. Then you will be on a good start to creating a nice Oracle twitter feed.

Retweet other peoples tweets. It will get  you noticed and help you build a following.

Twitter can be hard to sort the wheat from the chaff so I use a couple of tools.

• Prismatic
• Tweetdeck

Prismatic will go through tweets and pull just the tweets with links to articles and then server up the articles in a news reader. I love it.

Tweetdeck will allow me to track words or hashtags. For example I can monitor all tweets with my company name, Delphix, or during a conference like Oracle Open World I can track all the tweets with #OOW and #OOW15.

Start a blog if you can. I often use my blog as just a way to document stuff I’ll forget and will want to find later.

Comment on other peoples blogs.

Get on technical forums and participate. My favorite Oracle forum is the Oracle-L list.

If you write scripts to help you, share them on github and help other people with their scripts on github.

On linkedin, connect with everyone you can. If your company is small enough, just connect with everyone at your company. Connect with friends, colleagues, customers, everyone you can.

photo by  Daniel Iversen

photo by neliO