ukoug

UKOUG 2011: Using your Database as a Fileserver

One of the coolest things in Oracle 11g and onwards is a functionality called XDB Repository Events. Most of you probably know that based on XMLDB functionality in the database, the database also can be used in a File server kind of way by enabling the XDB Repository HTTP/FTP or WebDAV functionality via DBMS_XDB. XDB

Read More…

What Have I Let Myself in For! – UKOUG this year

One of my favourite Oracle happenings of the year is fast approaching, the UK Oracle User Group technical conference {see/click on the link on the right margin}. I’ve blogged before ( like here, last year) why I think it is so good.

I try and present at the conference each year and I go no matter if I am presenting or not.

However, this year I think I might have got myself into trouble. I put forward 3 talks, expecting one or possibly two to get through. One on Index Organized Tables, one on IT disasters and one as an introduction to database design – I’ve moaned about it being a dying art so I figured I should get off my backside and do something positive about it. Each talk is in a different stream.

UKOUG Agenda

As in previous years the UKOUG allows you to create a personalized agenda for the upcoming conference. To give you a flavour for some of the excellent presentations and to encourage you to register I’ve reproduced mine below, as with previous years this is a ideal wishlist, I’ll probably “die” halfway through various days and [...]

RAC and HA SIG meting Royal Institute of British Architects September 2011

I have been looking forward to the RAC & HA SIG for quite some time. Unfortunately I wasn’t able to make the spring meeting which must have been fantastic. For those who haven’t heard about it, this was the last time the SIG met under its current name-as Dave Burnham, the chair pointed out in his welcome note.

RAC & HA SIG is going to merge with the management & infrastructure SIG to form the availability management and infrastructure SIG, potentially reducing the number of meetings to 3 for the combined SIG. This is hopefully going to increase the number of attendees and also offer a larger range of topics. I am looking forward to the new format and am hoping for a wider number of topics and greater appeal.

Partly down to the transport problems that hit London today (Victoria Line was severely delayed and apparently overground services were impacted as well) the number of attendees was lower than expected.

The following are notes I have taken during the sessions, and as I’m not the best multi-tasking person in the world there may be some grammatical errors and typos in this post for which I apologise in advance.

Support Update-Phil Davies

The first presentation was Phil Davies’s support update which provided the usual good overview of what is currently relevant in Oracle support. My personal highlight was the fact that you can limit the number of child cursors per statement via an underscore parameter. This worked will for his customer who had to use CURSOR_SHARING set to FORCE.

Also there is an interesting problem related with Data Guard, the RFS process and overwriting of arbitrary files on the standby.

Plugging in the Database Machine-Joel Goodman

Joel delivered a very good presentation about monitoring the Exadata Database Machine. What’s great about Joel is his depth of knowledge and his ability to enrich the presentation with annotations both from the classroom as well as real life. If you haven’t bookmarked his blog yet, it’s well worth doing so from http://dbatrain.wordpress.com/ .

I personally have seen this presentation internally at a customer site, but still learned new things. Especially about the SNMP traps being routed back into the MS process on the cells, which can then be checked via cellcli.

Every so often the current metrics are flushed to disk, and move from metriccurrent to metrichistory. The metric history is kept for 7 days by default, and I think I’ll look at extending my monitoring solution to heave them into the database into a statspack-like schema.

Another interesting fact to know is that ADR is also available on the cells, including the adrci command with all its options.

Plugins for OEM 11.1 include

  • Infiniband plug-in
  • Cisco plug-in
  • ILOM (only for database nodes)
  • Exadata plug-in
  • PDU plug-in
  • KVM plug-in

Of course for these to work you have to install the agents on the database servers (only!). Once the agent is deployed, the plug-ins need to be deployed to the Grid Control infrastructure first before they are passed on to the agents.

The Exadata plug in requires the database server’s agent software owner to use passwordless authentication to the cell’s cellmonitor accounts. Also, the cells must be configured to report SNMP traps to Grid Control. I guess a thorough read of the plugin installation documentation might be needed.

I personally regarded the other plugins to be of less importance and decided not to record them here-I’m sure there is a white paper on Oracle’s website somewhere.

An interesting side note on the KVM which is missing in the x2-8 is the fact that you could still access the KVM if the internal Cisco switch failed. This is simply because the KVM does NOT go through the Cisco Ethernet switch, but rather directly connects to the corporate network.

High availability for agent monitoring a target is described in MOS note 1110675.1-a sure candidate for further investigation.

After Joel’s presentation we had a great discussion with Sally and Jason about disk failures in Exadata and the quarantine. In certain situations, if multiple disks fail only high redundancy can prevent complete disaster.

Also one should really be careful to not have negative numbers in V$ASM_DISKGROUP.USABLE_FILE_MB. If you do, it’s not an immediate problem, but an imminent danger as soon as a failgroup goes offline-there simply isn’t enough space for an ASM rebalance operation. Summary: you should not run your ASM mirrored disk group at full capacity to avoid trouble. Oh yes, and you should have at least 3 failgroups in a normal redundancy diskgroup.

I suggest you read Joel’s blog entry “mirror mirror on the Exadata” for a more thorough discussion of ASM mirroring in Exadata.

Exadata Storage and Administration-Corrado Mascioli

Corrado is a colleague of mine working in engineering on the same site. He has got great experience in patching Exadata and automating the process.

The cells are shipped with the software pre-installed, based on Oracle Linux. The most important accounts available are

  • root
  • celladmin
  • cellmonitor

These have various degrees of power, listed here in descending order.

Cellcli is the main interface to the storage cell allowing the user to perform administrative tasks.

The main cell processes are:

  • CELLSRV: mainly uses iDB to communicate with the RDBMS nodes and satisfies the I/O requests.
  • Management Server – MS
  • Restart Server – RS

Flash storage is something I blogged about earlier, see here:

Flash disks can be used either as Exadata Smart Flash Cache or Grid Disks, i.e. “ASM disks”. I haven’t created flash grid disks yet but suppose you would want to group the grid disks on a cell group to create failure groups.

David Burnham raised an interesting question about differentiating the flash cache in Exadata from the one available to the mere mortals, available with a patch or 11.2.0.x on Linux and Solaris.

The PCI cards you put into a database server are like another level of buffer cache, whereas the Exadata Smart Flash Cache is a) unique to Exadata and b)

Next Corrado explained the link between the physical disk, LUN, cell disk and grid disks. Especially the 30G taken away from the first 2 cell disks cause an interesting dilemma when it comes to the allocation of space for the DBFS disk group (former SYSTEMDG). For each cell, cell disks 3-12 reserve the last 30G on the innermost tracks of the disk for DBFS_DG.

The DATA diskgroup will by default use the fastest, outermost tracks of the disks, +RECO will take the middle of the disk whereas DBFSDG uses the innermost like I just said.

DBFSDG is mostly used for the database file system but also for the OCR and the voting files. DBFS looks like a normal file system for the end user.

I wonder if you could create a grid disk on a specific number of cell disks? I’d have to check the create griddisk command in cellcli …

All the settings are easily accessible with the cellcli commands list {lun,physicaldisk,celldisk,griddisk}.

The grid disks are visible to ASM using the CELL library (V$ASM_DISK.LIBRARY), and use the path 0//. By the nature of the technology all 14 x 12 disks are visible in V$ASM_DISK.

Each cell is its own failure group-which makes sense, given the fact that all disks share a single point of failure. Also worth remembering that since there is no storage array mirroring hence we resort to ASM redundancy.

Corrado shared lots of practical advice about creating grid disks and reconfiguring a storage cell using cellcli.

Panel Session-all speakers and The private cloud-Martin Bach

Well that’s me in the middle of the action-I hope someone else covers these.

Managing ASM redundancy-Julian Dyke

Julian started the RAC SIG in summer 2004, and surely had to have the honour to have the last slot on the current designation.

His opening theme has been a comparison of single threaded CPU performance for different architectures including Intel, AMD, SPARC, and IBM Power. Contact him personally if you are interested in the real results, it suffices to say that the 5600 Xeons are the fastest. I wonder if anyone has a recent Itanium processor willing to run the benchmark.

Interesting twist about a two node cluster and ASM tablespace creation with different allocation units-setting the AU size to 32M reduced the tablespace creation time for a 20G tablespace to a few seconds. There seems to be a lot of inter-ASM instance message exchange.

Following this Julian continued with the discussion of the ASM utility kfed to dump disk group metadata followed by a graphic visualisation about extent allocation and maintenance during ASM rebalance operations.

The nugget for today was to learn why the ACD uses 42 entries and ASM_POWERLIMIT used to be 11. 42 is self-explanatory. The power limit of 11 is a true classic-it’s one faster (louder), in honour to Spinal Tap.

There are a number of new features in 11.2 Julian mentioned which I covered in Chapter 8 of “Pro Oracle Database 11g RAC on Linux”. Of particular interest was the location of the 3rd voting file in stretched and non-stretched RAC if you used two different SANs for the first 2 failgroups. Oracle now supports the iSCSI/NFS approach for the third voting disks previously recommended for stretched RAC in “normal” RAC as well.

One of the features Julian didn’t mention was the location of the snapshot controlfile which since 11.2.0.2 also has to reside on shared storage-there is a note on MOS for this.

Oh yes, and then there was the demo about ASM normal redundancy which continued the discussion started at the last RAC SIG.

Summary

I enjoyed today a lot, met lots of interesting people and had many technical discussions about all sorts of things. One thing I’m looking forward to is the change of the SIG format, especially hoping for more attendees to make it even more attractive.

Unfortunately, due to the change of date for the Management and Infrastructure SIG which I will now miss I couldn’t see Piet de Visser whom I haven’t seen since Birmingham last year. Maybe I need to get to work on the same site as he does for a few weeks to catch up properly.

UKOUG Oracle Conference agenda now out

I just wanted to drop a quick post to say that the agenda for the UKOUG annual conference is now out. You can check out the schedule here.

They seem to have dropped the TEBS (Technical and E-Buisiness Suite) out of the title, I think because last year the UKOUG staff kept getting asked if it was the annual Oracle conference they knew and loved from prior years. And of course it is. (Other “application” sides of the Oracle world, like JD Edwards and PeopleSoft, have their own dedicated, named UKOUG conferences).

There is also a return of the Sunday OakTable stream. For those who have not come across it before, it is a chance to see some presentations by members of the OakTable in a smaller and more accessible room. ie you feel better able to ask the presenters awkward questions :-) .
I’m not sure of the exact details of registering for this part of the event but the agenda shows the talks that are happening (in fact, if you click on the “view the full 2011 agenda” icon on the agenda home page, it shows Sunday by default). I managed to get along to the OakTable Sunday a few years ago and loved it – I’ll be on the opposite side this time, I’m priviledged to have been asked to fill one of the slots.

As ever, the conference has a massive and wide-ranging agenda, with mini-streams like EXA(data/logic) and MySQL on Monday,APEX on Wednesday… The number of papers and the general quality that are submitted to the conference goes up and up each year and a lot of effort goes into not just picking well know speakers but also a mix of new presenters and ensuring topics get covered. It’s hard, but during the selection process sometimes there are 4 or 5 talks we know are going to be excellent but are all on the same or similar topic – some have to be dropped to ensure the breadth of topics is still covered. The number of slots a single person is allowed to have is also controlled, again to maintain space for a wide range of presenters and presentations. All in all, it is not a simple task and even now some tweaks are going on (to fill topic gaps, finalise the exact scope for a talk or to allow for people who suddenly find they cannot present anymore). You can rest assured though that, all in all, it will be an excellent conference.

UKOUG Presentation

I had a recent email enquiry after the slides from my talk at the technically excellent UKOUG Conference since problems with the website are preventing downloads currently. Reproduced below by way of a heads up is my email explaining that the slides are now available via the web, including to non-UKOUG members. Please feel free [...]

News from UKOUG 2010 Conference

Right now I’m sitting in the speaker lounge with Jeremy Schneider after hacking some RAC ASM stuff as a follow up to my last presentation. We were testing some failure scenarios but that’s a topic for another blog post.

Dan Fink cheated with his tiny blog post which was more like a twitt-long (and so did Christo.) so I thought to write something properly.

Monday started early for me — 6am. Quick run through my demos again and early breakfast. Registered before 8am while it was still empty and then joined Tom Kyte in the speaker lounge. We both had our sessions starting at 9am but Tom is a Pro when it comes to presenting — while I was taking the last minutes to go through my slides and do minor adjustments, Tom was calmly replying AskTom questions. Oh well, such is life.

My 2 hours presentation was a little slow and I wish the audience was a little more engaging but maybe it was just because all the locals hit the hibernate mode following “extreme” cold weather and didn’t quite wake up after the weekend (of course, there is not chance that it was bad presentation material or speaker…. no, no!). This was the same presentation as I’ve done at the OpenWorld but I included demonstrations of 11gR2 Grid Infrastructure and that was the tricky bit. In the end, everything pretty much worked with one small surprise. My last demo was troubleshooting of startup and I decided that I will screw up 3 things and troubleshoot online *first* time. I.e. I decided deliberately not to practice it. The latter wasn’t very smart as I had less then 10 minutes left. After few minutes of shame I had to move this demo in the list of homework. :) The good news, that I did go through my last slides briefly and I wanted to be brief there as Frits Hoogland was covering this area in more details later that day in his own session.

Exhausted after my session (and slightly disappointed by inactivity of the audience), I was very hungry but decided to wait until the official lunch time so ended up in Graham Wood’s session on some hidden free gems along with Christo and Jeremy. We did have a plan to switch to the Exadata round table and did just that. I’ve got a little disappointed with the round-table format (as I’m writing this — discussed it with round-table moderator, Joel Goodman, here in the speaker lounge and we agreed on this) – it was more like a presentation without slides with introduction to Exadata for folks who don’t have any background knowledge. It should have been a presentation while the round-table should have been left for the folks with experience or knowledge of Exadata technology.

Having been late from Exadata round-table I was late to lunch. This means I was late to the next session and managed to sneak in the last 10 minutes for Frits Hoogland‘s Oracle Clusterware 11gR2 In-depth. It looked like the audience wasn’t very active during his session either. When I spoke to Cary Millsap later, he also mentioned that it was somewhat a struggle to get the audience to laugh so he had to leverage his special jokes from the reserve list. I might borrow some of them in the future. :)

Next I went to the Tanel Poder‘s presentation on Exadata migrations and related performance tuning. Very insightful as you can expect from Tanel. He also confirms that Exadata performance rocks but it can be tricky to run stable. I think our experience was somewhat better with stability except early months when lots of issues were not fixed.

Afterward, I went to Cary Millsap‘s presentation on reading 10046 trace files. My intention was not to learn the subject that I was already familiar with but to learn how Cary can present this topic in his new style (with very few words on the slides). Turned out that he did put trace content on the slide but it was interesting to see how he emphasizes what he really wants to talk about in the 20 lines of code on the page. I will borrow this for my future presentations.

The final session of the day was Julian Dyke’s replication internals. It’s been a while I wanted to dig into replication deeper so it was a good move to go there. However, after such an active day (and night), I was struggling to stay awake even though my brain was desperately trying to keep up. The good news is that Julian have very well illustrated slides so I can always get back to it.

That evening, we had OakTable dinner. It was, of course, at the Indian restaurant. I admit I abused that place and barely was able to walk after dinner and struggled to consume anything more that evening. Still, nothing stopped us from hanging in Tap & Spile until almost 2am and even catch the last order of scotch at Jury’s. That was another abuse of the night especially that I had a presentation to deliver next day. Fortunately, the next day consequences were very mild but that evening brought me the idea of a demo for my presentation (thanks Christo) and most of Tuesday I spent getting this demo ready. It worked very well but I will need to improve few items to run it faster.

It’s already Wednesday as I’m finishing this post now. The night was lots of fun and it was long and… very late. I recall that the most bizarre idea of that evening was robbing a bank (don’t ask how we got there… it was not my fault). I didn’t really pay attention when I was back in my room and crashed but I see my last email from the phone was sent at 4:25am (Hi Doug!).

I missed the presentation on marrying Grid Control and Nagios — very interesting topic for me as some of our customers happen to use both. I struggle to understand why one would want to integrate these tools but that’s why I really wanted to see it. Oh well, I have to review the slides offline and I’ve met the author the evening before so I could always contact directly (thanks Eter!).

Half more day to go. Still struggling to decide whether I should go to Julian Dyke‘s presentation on memory (I know Christo did his this morning but it was way too early for me) or to the session on RAC Server Pools by Bob Mycroft (somehow, his name is associated with Windows – is it just me?).

Oh… I completely forgot to mention that the highlight of Monday night was Doug Burns shaving ceremony and finalizing it at Tap & Spile. I have captured some videos but they need some post-processing before I can publish them. Another highlight was watching photos of previous UKOUG conferences I attended and I specifically liked one photo that was not supposed to be there! It won’t make sense to you my dear reader but please forgive me and ignore it — it’s meant only for one of you. ;-)

Pythian at UKOUG Technology and E-Business Suite Conference 2010

Hello Birmingham!

It’s past Sunday midnight and I’m stuck in my room in the last couple hours finishing my slides for my masterclass tomorrow. Turns out that I’m presenting the very first session of the conference at 9am. I wish there is a keynote instead so that I could grab one more hour of sleep (it’s going to be deep into the night back home in Canada). Strange that the keynote was moved to Wednesday — I hope UKOUG has really good reason for that!

My two hours masterclass will start at the same time as Tom Kyte’s a-la keynote session — what a competition. On the other hand, there is no other sessions in server technology so I expect that folks without interest of database development will automatically end up in my session. I’m in Hall 5 – quite large room. Is it the second biggest room after the Hall 1?

I will need to work hard to keep the audience… maybe I shouldn’t plan for any breaks to make sure I don’t let folks slip out to the next sessions like James Morles’ Sane SAN 2010 or Jeremy Schneider’s Large Scale ASM.

My masterclass is based on the slides that I presented at the Oracle OpenWorld few months ago which, in turn is reworked session on Oracle Clusterware internals that I’ve done number of times as long session with demos. I thought updating this material to 11gR2 would be easy… Boy was I wrong!

11gR2 Grid Infrastructure has changes so much that it took me much much longer to get something sensible ready. I also had to limit the scope a bit as Grid Infrastructure has become so much more complex than older pre-11gR2 Clusterware. (stop complaining Alex!)

Anyway, everything is ready now and demos look reasonable. It will be a bit rough doing it first time – I’m sure I’ll stumble few times but fingers crossed we get to the end timely. I actually hope to finish early and allocate a bit more time for Q&A and potential ad-hoc demos at the end. But enough about me…

Who from Pythian are at the UKOUG conference this year? In additional to myself, it’s Christo Kutrovsky, Daniel Fink, Paul Vallee and Andrew Poodle. Christo, Dan and myself are presenting, Andrew is helping organization of MySQL track as a MySQL SIG Chair and Paul… well, I’d say Paul is a slacker so he is covering the beer tap to pay up! :)

It’s close to 2am – gotta get some sleep before tomorrow. Few words against Jurys Inn Hotel this year. It’s the first year I’m having so much troubles here including no early check-ins, not working phones, no internet in two rooms (I had to switch twice!), and somewhat unfriendly stuff this year. Has hotel management change since last year or what? Will consider another hotel next time I think.

Oh… and it’s indeed bloody cold here! So cold that it seems to impact the amount of girls-who-forgot-their-skirts-at home at the Broad Street. This unusually cold weather does impact travel plans of other conference speakers and attendees. Doug Burn seems to have been delayed for like a day and barely made it to have a pint at Tap&Spile – I wish I could accompany the crowd there until late but thanks to the awesome schedule (and unfinished state of my presentation, to be fair) I had to miss some of the fun.

PS: I have another session on Tuesday — Analysis of Oracle ASM Failability (should be Fallibility I guess but I’ll keep it misspelled simply because I can!). If anybody wants to catch up for any reason (like buying me a beer) — text me at +1 613 219 7031. iPhone doesn’t work with data-plans here for unknown reason so no twitter/email on the go.