"John VanSickle" <evi### [at] hotmailcom> wrote in message
news:47191438@news.povray.org...
> At the school where I am the computer skills teacher, the students in one
> of the science classes were doing projects on the chemical elements. Each
> student was assigned a chemical element, and told to research that element
> on-line and in the school library.
>
> The student who drew the element mercury thought it would be nifty to have
> samples of the element in his project.  As I understand things, his father
> works in a field where mercury switches (which are little glass bulbs with
> mercury and wires in them) are used.  This youth brought several of these
> to class, almost certainly without his father's permission.
>
> Some of these little capsules were broken in the school building,
> resulting in mercury contamination in one of the boys' bathrooms, the
> locker room, and my classroom.
>
> Resultantly, a HAZMAT team was in the school today, tearing up the
> carpeting in my room, and doing their work in the other places as well. My
> classes were relocated to another room, where the air conditioner was off.
> Although this is middle October, this is also central Florida[1], so the
> room was uncomfortably warm.  The systems on which the educational
> software is installed are still in the room being decontaminated, so the
> students participated in their Accelerated Reading program for the day.
> They sit and read for half an hour.  Most of them hate it.
>
> The kid who brought the stuff to school was suspended for two days, and I
> was asked why I didn't notice kids playing with mercury in my room, by
> people who seem to have forgotten that if kids really want to hide
> something from the teacher, it requires Superman's X-Ray vision to catch
> them.
>
> So, life at school sucked this week.  How's life for you?
>
> Regards,
> John
>
> [1] Yes, those of you who live near The Villages in Florida may have seen
> this on the news.

When I was a student, our teacher brought mercury to the classroom in an 
open beaker.  We could dip a finger in it and feel its heaviness press 
around your finger.  While handling it, her ring finger dipped in the 
mercury and her gold wedding ring came out "silvered".  Whoops.  She dropped 
a little on a piece of aluminium and then shook it off.  Fresh white oxide 
grew on the bare aluminium as though it was magic fast-gro hair.


Here's my week:

 I have been attempting fix/salvage of an Oracle database that I now believe 
was incompletely upgraded in the past.  It's a 9.2.0.7 database and has 
worked without problems for ages.  It is the Enterprise Manager repository 
for managing about a dozen 8i and 9i databases, plus it has the Recovery 
Manager catalogues for backups of another three 10g Databases.

It all started when we had had problems with peripheral hardware on the 
server.  We rebooted and power-cycled the server to clear it.  I manually 
shut Oracle down cleanly.  On startup, all Oracle restarted normally (2 
databases, one listener, one intelligent agent, one EM console, one iAS 
1.0.22.

That night, the full database export failed with an ORA-0600 error.  An 
ORA-600 is the dreaded "internal error", and generally means "bug".  It 
failed the next night too.  And the next.  Otherwise the DB worked fine.  It 
just wouldn't export any more.

The message arguments that came with the ORA-600 indicate the exception 
occurred in Java
routines.  Oracle's Metalink site lists many separate bug instances and 
patches for this Java-specific problem, but also had a remarkably easy 
workaround:  Bounce the database and this problem will go away.

We bounced the DB.  The export still breaks.  Bounce again.  Repeat ad 
nauseum.
Nope, that workaround don't work on this DB.

Ok, so this DB is at version 9.2.0.7.  The terminal release of 9iR2 is 
currently at 9.2.0.8.  That led me to the first tactic:

First tactic:  Update version to 9.2.0.8 in hope of clearing the bug.  If 
not, well, we can get support from Oracle once it is on the terminal 
release.

I downloaded the 9.2.0.8 patch set and began updating.  The install phase 
went well.

While running the "catpatch.sql" to update the data dictionary objects, 
several errors threw up here and there without an obvious reason: 
"ORA-1460: Unimplemented or unreasonable conversion requested.".  The Error 
Codes manual says of ORA-1460 in so many words:  "You stuffed up a 
conversion format string in a TO_DATE, TO_CHAR or a TO_NUMBER function. 
RTFM and get it right, you idiot."

But weirdly, this error threw on statements like "create public synonym", 
"grant select on ... to PUBLIC", "create or replace view" and "drop table" 
I watched as the patch script ran:  It made no sense.  I was studying and 
pondering it while the patch SQL scrolled by in SQL*Plus, when...

BAM!  "ORA-03113: End-of-file on communication channel."  It happened on one 
of the statements in the catpatch (actually in catjob.sql called by 
catpatch) : it was a "grant select on ALL_JOBS to public with grant option". 
The remaining several hundred of the patch's SQL statements all returned 
"ORA-03114: Not connected to Oracle.".

I tracked down the trace file for the dropped SQL session.  The HP-UX 
operating system had killed my SQL session on a Floating-Point Exception 
(signal 8).  Nice.  Oh, and it occurred in a Java subroutine.  Did I mention 
Java?

For a while I tried to push the semi-patched database through to a usable 
state, but the session aborted on too many statements, plus those !#%$ 
ORA-1460s that seemed unrelated to the operations they threw on.  I 
suspected something wrong in the data dictionary made Recursive SQL (a term 
for Oracle's internal SQL on the data dictionary, that services your SQL 
request) throw a 1460.  But normally when a Recursive SQL error breaks your 
SQL statement, it tells you that's what broke.  This wasn't doing that.

THEN, I finally "saw" the banners for the very first time as I logged in 
*yet_again* with SQL*Plus.  Roughly it looked something like:

Connected to Oracle RDBMS 9.2.0.8.0 Enterprise Edition 64-bit -- Production
With Data Mining, Partitioning and Spatial options

JServer Release 9.0.1.4.0

SQL>

WTF was that?!?

The Java Machine in this database is a 9i Rel.1 version and the DB is 9i 
Rel.2!  How the h3ll did that happen?!  I checked out a 9.2.0.6 DB somewhere 
else.  Sure enough the banners including JServer are all 9.2.0.6.0.


Further pursuit of upgrade was pointless, so I shutdown the DB.  Blew it 
away, and the software home and the installer's inventory.  Restored it all 
from tape, started the DB and did media recovery to a point-in-time just 
before the abortive update.  Sure enough, RDBMS 9.2.0.7.0 and JServer 
9.0.1.4.  It must have been like this for three years at least!

Second tactic: Data protection Strategy:
In case this DB ultimately has to be thrown out, I wanted to preserve the 
RMAN catalogs and Enterprise Manager Repository.  I really don't feel like 
rebuilding all that.  The full export was failing because of a Java bomb, 
but maybe if I export only the RMAN user, I could extricate some data before 
the caboose heads over the cliff.

I made an export par file.  I started the export.  The server fell to its 
knees and nearly died.  More accurately, "exp" ran, printed no messages, not 
even a banner.  It sucked 100% of one CPU and proceeded to eat memory like 
carnival ice-cream.  Within one minute, exp had a 1.26 GB resident memory 
size, disk I/O was at 100% bandwidth as "vhand" swapped everything that 
wasn't "exp"out to disk, including both databases.  It took two minutes just 
waiting to pull a new command prompt, or a login prompt, or anything.

I couldn't get rid of it:  The runaway export cheerfully ignored my <Ctrl-C> 
(Interrupt) <Ctrl-\> (Quit) and then repeated "kill -9" signals.  I 
abortively shutdown the database it was supposed to be exporting.  (I gave 
up on the normal shutdown after waiting for three minutes)  It still 
wouldn't die.  I set up a shell command loop to send it a kill -9 every two 
seconds.  Finally I also shut down the OTHER database to try free enough 
memory to get functionality back, and at last the "exp" process died, on 
about the 23rd kill-loop iteration.  It released 1.2 GB of memory and the 
system was at last responsive again.

Phew!!   But -- it meant the death of hopes of preserving data to carry to a 
new database.  :-(

Third tactic:  Uninstall and reinstall the JServer to bring it up to the 
proper database version.

A colleague had made up an SQL script in the past to do just this on another 
DB, and he gave it to me.  This didn't work for the same reason as before. 
The SQL connection just died as soon as it touched any Java-ish object.  So 
the JVM is even too faulty to drop.  And AGAIN, those ORA-1460 messages!

Seems much more is wrong with this DB than just Java.  It must have gone 
through a botched upgrade from 9.0.1.4 and god knows what else could be 
wrong with it.  It's a miracle it worked at all for this long!  I blow it 
all away.  Restore from tape.

Fourth tactic:  Do a *full* 9iR2 base install (9.2.0.1.0) and run the 
9.0.1 -> 9.2.0 migration procedures over the database.  If that's ok, patch 
up to 9.2.0.8.

Where the hell did our 9.2.0.1.0 software for HP-UX go??  Dammit.  I had to 
download all four disks worth from Oracle Technet.  Took about a day. 
Still, thank god the Internet makes this possible.

That's brought me to Thursday afternoon and I could do no more with it this 
week.  On Friday I had to prepare for a planned cutover of another 
application to its Disaster Recovery site database.  The primary building 
was being knocked off power supply (also my office)

Thursday night: A callout.  Friday night: A callout -- the application is 
extremely slow on the D-R database.  Today (Saturday) more work on the D-R 
callout, but finally resolved.


As for the "Fourth tactic" install on the 9.2.0.7 / 9.0.1.4 dudderbase, it 
will come next week.  Watch this space.

If that doesn't work, the fifth tactic will also be my last one:  Trash it 
all and build a completely new database, Enterprise Manager configuration 
and RMAN catalog.  A long job.  The most recent valid export of RMAN and 
Enterprise Manager data will be old and on tape, but might be usable.

-- 
Brian

Post a reply to this message