Yes, this definitely sounds like the sort of thing Oracle is supposed to avoid. I suppose if we had archive logging turned on we'd have one or more extra logs being written to and more chances for the needed info to make it onto disk, but we'd have more chances for failed writes too, so I'm not sure whether that's a net plus or minus on this particular point.
I won't be opening a TAR on this. In order to do that I'd have to leave the system untouched, and I've already done as much recovery as one can do so the evidence has been spoiled. And today we'll be reinstalling the system with RedHat 7.3 so I can install Oracle with all the proper patches, so then the evidence will be gone. This is the most powerful system we have, and the site's currently on a lesser system and really chugging. I can't wait around on this.
I did post about this to the forum on Metalink, so if someone from Oracle is interested in knowing that it happened, the record is there for them to see. There were plenty of people with this problem posting there, but most of them had come about it differently. They'd experienced a full system failure, had to restore from cold backups, and found that their backup was hosed because Oracle wasn't really shut down when they took the backup like they thought it was.
On the whole, I'm actually pretty happy with the nightly-dump backup strategy. There are plusses and minuses to cold and hot backups; with the dump method there's a high likelyhood of having to reinstall Oracle to recover from a catastrophic failure, which the cold backup method avoids. But IMHO the likelyhood of either AOLserver or Oracle failing to start back up some night is greater than the likelyhood of having a catastrophic failure (I think this is the first one I've had in 3.5 years). I think if I add archive logging, which saves me from losing the day's changes, we'll be in good shape.
