Digital Medulla

Well, I passed.  92.6% on the RHCT portion and 93.8% on the RHCE side of things.  I am officially an RHCE now.  Of course, the best part is that the studying and hard work paid off (and that I actually *will* be getting the reimbursement).  I spent Friday night celebrating (to the point that I could barely concentrate on the game), had our Halloween party on Saturday, and generally just kicked back and treated myself to a nice reward yesterday.

I’m talking, of course, about Fallout: New Vegas.  It had been sitting on my bookcase since release day, just waiting to be opened.  I promised myself that I wouldn’t play it until after the exam was over, and I actually held out.  It’s been fun so far, but I’m only about three or four hours in and just reached level 6 last night.  I haven’t found any particularly awesome weapons or armor yet, but I have already made enemies out of the Powder Gangers.  It’s nice to be hated by virtual escaped cons with a penchant for dynamite.

I’ll be taking the RHCE in approximately 30 days…  October 29th is the slated test date, which means I get to drive to Raleigh on the 28th.  Before you say anything, the only other nearby option was Tyson’s Corner, and we all know how I feel about the metro DC area…  Quite honestly, I’m rather excited to take it.  I look at it as official recognition of what I can do, and certs usually help your case. 

So I’ve decided that I need to make myself a ‘cheat sheet’ based on the currently-listed exam objectives, which I’ll be doing tonight.  Don’t worry, it’s not designed to be an aid during the test, but rather just a one-page overview of what I need to know.  I’ve also downloaded some example tests and a few reference documents from some other blogs that have talked about the exam.

The only thing I’m not liking is the cost of the exam…   But at least I get reimbursed when I pass it.

New theme, eh?  I need to work on customizing the graphics, but that’s really a minor issue.  The old theme, while interesting, was actually getting a little annoying on the old optical receptors.  Dark is good, but the neon pseudo-aurora was making it look a little to 80′s.

Krelian had some issues about a month ago after a storm, and I ended up having to do a major upgrade.  We’re now running with a quad-core @ 2.3 GHz, 4 GB of RAM (no change there), and an Antec 500 watt PSU.  After a little flakiness on the power front (I think the molex was loose on the MoBo – things got better after I yanked and re-seated it), we’re good to go.  I think on the next round of updates I’m going to drop the current 80 GB IDE boot drive in favor of a similarly-sized SATA SSD.  <fred_rogers>Can you say ’10 second boot time’, children?  I knew you could.</fred_rogers>

Things are progressing mostly well on the work front.  The monitoring migration is largely done and so now we’re working on moving people from our old ‘backup’ platform to the new one.  Why the air quotes?  Well, the old platform is a SAN hooked to a tape robot, but due to how the thing’s configured, it’s really more like a glorified USB disk than a real backup.  The new system is marginally better, but I’m not completely thrilled with it yet.  At least it does incrementals instead of a full nightly every night…

We’ve also moved our best and most promising NOC tech up to Engineering.  Sadly, he mostly handles provisioning work, but it’s still a far cry better than having him work on relatively low-level tickets.

The house is coming along nicely, though it too still needs work.  That’s the joy of home-ownership, isn’t it?  It’s an endless battle between you and the house.  The other week some friends and I built out a large-ish platform in the crawlspace to use as a storage area.  It’s made of pressure-treated lumber and OSP, so it should be fairly rot-resistant.  Not the way I originally wanted to do it, but it was considerably easier than using cement (and far cheaper to boot).

Am taking a brief vacation in the beginning of next week.  We’re going to Charleston for a couple of days, during which time I’ll actually unplug for a while.  I wish I could say I was looking forward to it more, but I just really find it hard to be excited.  There are a number of reasons for this, including but not limited to 1) the fact that I’ve already been there a couple of times, 2) it’s a 7.5 hour drive on I-95, and 3) it’s going to be expensive.

In other news, I’ve been playing Arc Rise Fantasia lately.  It’s a solid game, but the voice acting is really distracting.  Would it have hurt to have people who put some passion (or at least skill) into their lines?  Oh well.  That’s the only bad thing I can say about it, so I guess it’s doing pretty well so far.

—————————————————————————

P.S. – Holy shit.  I haven’t updated since January.  I really should try to do this more often…

This morning I helped a customer change out their old firewalls for a new pair.  In the process this hideously broke their site.  We had attempted this conversion before with no success and had to revert – it was broken enough that two of our other engineers couldn’t figure out what was wrong.  I fixed it.  What follows is a brief analysis of the process.

In the error logs on the apache server I noticed numerous failed connections to the database.  Okay, that makes sense.  A command-line connection also failed.  The web server could see the database server, but it wasn’t getting any responses.  WTF?  The database didn’t show anything out of the ordinary in its log, so I started some basic network diagnostics.  Traceroute from web to db worked, but traceroute wasn’t installed on the database server.

I tried to install traceroute, but discovered that outbound connectivity from the database server to the world was blocked.  We had the firewall guys open up ICMP and set up the necessary NAT rules.  Once ping was working, I tried to install again only to learn that DNS was also being blocked.  Magically, after DNS was allowed, MySQL began to respond to requests from the Apache server and the pages loaded normally.

So what happened?  Well, it turns out that by default MySQL attempts to perform a lookup on new client connections¹.  Since DNS was blocked it couldn’t do this and was just stalling on any new connection.  We probably could have corrected the issue by modifying the hosts file on the database server, but it’s probably easier to set up and less maintenance in the long run to just allow outbound DNS requests.  Alternatively, we could have reconfigured MySQL to not do this (using the –skip-name-resolve option), but that would mean that any GRANTs would have to be purely by IP address.  I can verify that the users in mysql.user are only entered by IP, but that doesn’t mean that something in one of the tables isn’t.

Morals of the story?  1) Always allow DNS requests to go out.  2) Sometimes it really is the little things that mess you up.

—————————————————————————————————————
¹ – http://dev.mysql.com/doc/refman/5.0/en/dns.html

Fuck my life.

I got a call this morning from our attorney stating that closing would have to be delayed for at least two weeks because the seller, well, is being a pain.  She’s in Dubai right now and can’t get the deed notarized for the closing.  Well, actually, she could get it done, but the US government / state of Virginia won’t recognize the ‘legal authority’ of a Dubai notary.  Why do we have notaries again?  They’re an outdated and needless waste in this society.  And, again, she could get it notarized if she could get to the American embassy, but apparently there’s some trouble with doing that.  I understand that things like this happens every so often, but would it have killed her to think of these things before she went out of the country?  The worst part?  She can’t / won’t hand over power of attorney to her agent or son so that we can just get this finished.

So now I’m stuck in this crapsack apartment for another two weeks, I have to reschedule all of my utility moves, and my taxes will have gotten pooched beyond reason.  Oh, and I won’t even really have time to unpack before I have to leave for Atlanta.

I was angry enough today that I literally wanted to break things and inflict bodily harm.  Even now, some ten hours after getting the news, I can’t really bring myself to concentrate on much of anything, be it gaming or writing.  I still kinda want to go out and commit ultra-violence on a grand scale…  Of course, the worst part will be telling Jen about all of this.  I can only imagine how she’ll take the news, and I’m not looking forward to it.

The idea of a breach of contract lawsuit is sounding mighty appealing, though I’m not entirely sure what all legal recourse I’ll actually have under the circumstances.  Given that tomorrow is pretty much a holiday in and of itself, I doubt that I’ll be able to talk to the attorney again before next week.  I’ll keep you all posted.

I took the opportunity to upgrade both of my *nix systems to Ubuntu 9.10 this past weekend.  While the laptop upgraded (mostly) flawlessly, I can’t say the same for my server…

On the laptop the only issue I had was caused by my (foolish) use of the proprietary nVidia drivers, which naturally flaked out on me when they encountered the newer kernel.  So I ran the installer in its uninstall mode, did a quick ‘sudo apt-get install nvidia-glx-185′ to put the Canonical-provided drivers on, and I was back in business.  I should mention that this upgrade was done using the Alternate Install CD & the CD-based upgrade script so that I wouldn’t contribute to excessive load on the mirrors.

With the server, well, a funny thing happened.  I tried using the same approach, but when it finally came time to reboot, the system just crashed.  Actually, that’s a little disingenuous – it tried to boot and failed.  Every time it tried to start, it failed to mount /dev/sda1 (a.k.a. /) and began a forced fsck.  Fsck died after about five minutes (roughly 3% of the way in) and dropped me to a recovery console.  A couple of hours later I gave up on the drive and replaced it with a brand-new 320 GB SATA drive.

I then spent three more hours banging my head off of my desk trying to get the system to boot again…  With the new drive, the OS would install, but it wouldn’t boot.  Turns out that I had forgotten to place the SATA drive higher in the boot priority in my BIOS.  Yea for feeling dumb…

Anyway, Krelian is back with 64 GB for /, 250 GB for /home¹, and the old 250 GB IDE drive still mounted at /home/me/media.  Maybe in another year I’ll replace the second IDE drive with another 320 GB SATA one (if they’ll still be making consumer-grade drives that ‘small’ in a year) and finally get around to configuring RAID on my home server.

In the mean time, I’ve got enough backups of my data that I’m not terribly worried².  Turns out the only things I lost by the original drive dying were:

1. My bookmarks -> mostly recovered from an old backup
2. My mail -> archived on the Gmail, so I didn’t lose too much of it
3. My XP VM that I use for work -> this can be rebuilt in little time.

Anyway, that’s about it for now.  I have a few more apps to install to get everything fully back to normal, but Karmic has been good to me so far.  Also, I’m extremely happy with & pleased to have support for MySQL back in the newest version of Amarok, but more on that in another post.

Later.

————————————————————————
¹ That’s /dev/sda1 and /dev/sda2 for those of you keeping track. /dev/sda3 is my 2 GB swap partition.
² I keep one copy on an external USB hard-drive, one copy in an archive folder on another physical parition, and one copy uploaded to an off-site server. Now if only tapes were more affordable…

Over the last few years I’ve somehow managed to get myself involved in a number of social-networking sites, and it’s finally occurred to me that they’re all pretty much useless.  With that in mind, I’m bringing everything here.  More details when I’m not at work.

After many years of having the same old laptop, I have finally broken down and purchased a new one.  Granted, I’ve been wanting a new one for a couple of years (and needing a new one for about 1 year), so it’s not that bad.  I dropped roughly $400 after tax and got a system that will probably last me another five to seven years. It is, unfortunately, a Compaq, but the price & feature set was right.

Now, instead of a single-core P-III 700 w/ 512 MB of RAM, I’ve got a dual-core Athlon-X2 x64 @ 1.00 GHz w/ 3 GB of RAM and a built-in nVidia 8200M.  Sure, it’s not the best graphics chip, but it’s more than enough for what I’m doing (which is to say, not gaming).  Also, this system can actually run KDE 4 AND open up multiple spreadsheets at once!  Yea!

Oh, and there was a bit of difficulty in setting up this system…  See, it has an AUO 1366×768 widescreen LCD display, which, even though the vesa drivers get along with, doesn’t play nice with the official nVidia Linux drivers.  I spent roughly 10 hours going through forums, editing my xorg.conf, and trying different distros before I found the actual fix.  Not surprisingly, the real fix is pretty damned simple…  Just drop the line ‘Option “ModeValidation” “NoTotalSizeCheck”‘ into the xorg config (and possibly ‘Option “ModeDebug” “True”‘ as well) and restart X.  After that, replace the current nVidia driver with the 190-series beta driver & good times are had by all.  Of course, this also means that I’ll get to sleep before 4:30 tonight.  More yea.

In keeping with tradition, the new laptop is also named after an RPG villain; like Krelian, Sephiroth, Grahf, an AM before, this new system is known as Gabranth.  Although you’d think I’d name my systems after female characters…  Perhaps I should rebuilt it as Myang?

Anyway, the old laptop is destined to have Windows XP Professional SP3 put on it (that’s the latest windows build it can really run) and will be sold off either on eBay or Craigslist to help defray the (semi-unnecessary) expenditure of a new workhorse.  I’ve got some other parts that I can put up there as well, so I should be able to recoup about 1/2 of what I paid (assuming that I can get $75 for the old system – if I drop Office XP on there, it’ll probably help greatly).

A few months back, one of my former colleagues and I configured a pair of VM host boxen to use jumbo frames and then attach to an NFS export on another server (also configured with jumbo frames) so that they can use that as a datastore.  Since that time, the NFS server has crashed hard twice – once because of an integrated NIC failure that borked the PCI-X bus and once for an unknown reason.  I say unknown becuase the manufacturer insisted that in order to replace hardware, they would have to have the system rebuilt with RHEL.  The server was originally configured with OpenSuSE, but, apparently, that’s not a ‘supported’ disto…

Anyway.  As it turns out, NFS doesn’t get along with jumbo frames.  Also, jumbo frames are only unofficially supported in ESX 3.5.  Yup, that’s right – use them at your own risk.  Now, why did we do something like this on production equipment?  Well, I honestly don’t know, and it wasn’t my idea.  Sure, people say that all the time, but I actually did voice my concern about this before it was set up.  In this case, however, I’ll just say that this former colleague has a tendency to produce off-the-wall, technically impressive, but very difficult to support solutions.

So here I am, VPNd into my corporate network, manually trying to recover a guest filesystem for a customer instead of doing what I should be doing on a Sunday morning – buying groceries or playing Wii.

Drag me to Hell was better than I expected, though I’ll admit that the ending was a little predictable.  Still, it’s worth seeing.  I’ll give no spoilers here.  I will say this though: the audience needed to have some manners smacked into them.  We had the privilege of watching the film in a small theater (perhaps 50 seats) with an audience of mostly teenagers.  These little twits sat and laughed at the most random and inopportune moments in the film, which is something that I just don’t get.  Finally I got sick of listening to it and waited for the time to strike…  One ‘Tyler Durden’ laugh later and the morons stopped laughing for the rest of the film.  Mission accomplished, time to finish watching the movie.

On the way home from the movie, we got to see a truly spectacular lightning storm.  I’m not normally given to hyperbole (or even mere exaggeration), so understand when I say that this was spectacular, it really was.  There were lightning strikes that seemed to last for upwards of two seconds, multiple strikes back-to-back, and some bolts so intense that they provided nearly daylight levels of illumination for their duration.  Sadly there wasn’t too much rain accompanying the lightning as we could really use that right about now.

Also, I finally found a good book about Nagios & managed to secure a copy of Cory Doctrow’s* ‘Little Brother’.  Once I finish reading it, I’ll have to post my thoughts.

——————————————————————————————————–

* See also BoingBoing.net