Technical

Last week was busy… making travel arrangements for this week’s trip to New York (technically Jersey) and some light analysis of AWR reports from exadata RAT runs and some heavy troubleshooting of a Solaris x86 RAC cluster with random node reboots. (I think I finally traced the node reboots to a kernel CPU/scheduling problem). I really did thoroughly enjoy my time in Africa despite being nowhere near Oracle software – but it feels good to be working on challenging cluster problems again!

It has been nine months since I’ve written here. Needless to say, a lot has happened!

First, my family was living in Africa for three months earlier this year while I did some tech work at an NGO hospital. Second, upon our return I decided to join the good people at Pythian. I’m not moving to Canada, although I will travel a decent bit as part of the company’s consulting group.

If you’re interested in the Africa trip, look at the Africa page. I wasn’t working with Oracle technology but it was still a very interesting, challenging and engaging project.

I thought I’d briefly share a few high-level insights. You might be surprised how well these lessons apply almost anywhere (even Oracle-related projects)!

Four Lessons from IT in Rural Africa

1. Understand the Fundamentals

Before I dive into this blog post, quick heads up for anyone attending UKOUG: on Tuesday only, I’ll be hanging out with some very smart people from the IOUG RAC Special Interest Group in the “gallery” above the exhibition hall. We’re ready to help anyone run a RAC cluster in a virtual environment on their own laptop. And if your laptop doesn’t meet the minimum requirements then you can try with one of our demo workstations. Come find us!!

Why Make Charts

I’ve heard Kyle Hailey speak on a few different occasions, and more than once he’s talked about the power of visualizing data. (In fact Kyle was a key person behind Grid Control’s performance screens.)

Want to get your hands on a key technology in both the Exadata Database Machine and the newly announced Oracle Database Appliance?

If you’ll be at OpenWorld – in just 11 days – then the IOUG RAC SIG is putting together a special event for you!  (You might have already heard about this on Twitter or from Justin at the OTN Blog.)

Every day from 9am to 1pm, find our table in the OTN Lounge (on Howard Street) and we’ll help you get an 11gR2 RAC cluster database running inside virtual machines on your own windows-based laptop. You can experiment boldly – if you make a mistake then you won’t have to start over; we can easily “reset” your virtual machines to any point.

One of my recent customers was a company with a somewhat large warehouse (around 60TB) on Oracle 10gR2.  The system was using RAC, though it was a fairly simple setup: two nodes, very large AIX LPARs, workload manually partitioned between them and somewhat evenly balanced.  The most important demand of their business is a large number of reports that must be generated every day from the warehouse.  These reports were beginning to take most of the day and consume a large amount of resources… and the current forecast is for dramatic data growth later this year.  So our project goal was to improve performance.

Lets suppose you are a DBA at a large company. You have some great developers, and they’re learning all about how to turn on full logging of their code through the 10046 database trace. They just learned how to use this data in summary form to find out – at a very detailed level – what’s REALLY taking up all the time during their big batch program which runs too long. They’re salivating over this trace data – but you work for a big company with security policies that can’t be easily changed, where developers rarely get any kind of shell-level or filesystem-level access to a database server. You WANT them to have the ability to profile their own database code… but every time they run a trace, you get dragged into a long email exchange to locate their tracefile and transfer it to a network drive where they can access it. We’re so close to a great situation… but this last part is such a drag!!!

Overheard in an IRC chat room (Freenode#oracle) this morning…

First of all, the RAC Attack deep dive at Collaborate went great – thanks to everyone who participated! The room was full (20 participants) and I got evaluations from about half of them. Here’s a summary of the eval results:

• 100% class met expectations, would recommend to others

• 66% easy to follow, could use skills in working environment

• 100% already familiar with oracle, 90% use oracle daily

• 0 negative reviews of instructor (phew!)

• 1 negative review of curriculum: said practice exercises weren’t relevant but training manual was still above average.

• 0 negative “comments”

There were several positive comments such as this: “I would recommend this class to others. This setup is perfect to pick up new skills and expose what ifs w/out worrying about pressing the wrong button.”

Over the past year or so I’ve had a number of conversations about running Oracle RAC on Amazon’s EC2 cloud platform. Chet Justice had suggested a long time ago that I try it, but I never quite found the time. Last fall at the Oak Table Symposium in Michigan, Jeremiah Wilton told me he hadn’t yet done it and I spent the last night scheming with Charles Schultz about getting started. But it wasn’t until this week that I finally found the time to try. (To be fair, I’ve been quite busy lately — our first kid was born on February 2nd, a beautiful little girl!)

Just thought I’d do a quick post on this one; came out of a conversation about a month or two ago.

We had a single-instance database running on a failover cluster (RHCS). A database link existed for a related database and the connection had to pass through a firewall. The problem was the firewall: it had a rule which only allowed connections from the VIP.

The database server has two IP addresses – a system IP and a VIP. Is there any way to bind the dblink to one specific interface? (Note: we would still like the system IP to be used for other traffic.)

I couldn’t think of a way for Oracle to do that. But we did find a workaround, of sorts (though not perfect) – by using the operating system route command.