Pants Full of Unix

Something verging on two months ago, I noticed that I wasn’t able to reach certain websites that I frequent (usually by way of RSS feeds) from my home Internet connection (a business DSL dry loop through Verizon). Some examples: Foobooz, riotclitshave (it’s a pun on right-click, save; shut up), Jeffrey Morgenthaler’s blog. What these sites have in common is that they are hosted by Dreamhost. The sites weren’t down (using my VPS as a proxy confirmed that without even needing to bother with things like Down for everyone or just me?), they were just failing to send packets back to me, semi-specifically.

I did a bit of research and found that not only did it appear that I was pretty obviously being explicitly blocked, other customers of my ISP were not. This irritated me more than a little bit, since I do like to believe that I have a reasonable idea of what I’m doing, I do take care of the security of my systems and networks at home (although I’ve been known to get a bit lax given that I do this shit all day at work), and I strive to be a very responsible network neighbor.

So I went looking for contact information for Dreamhost on their website… and found things sorely lacking. Web forms yes, one that applied to my situation, not so much. Phone numbers, no. Actual email addresses, no. This didn’t look good, but I went ahead and filled out a web form and emailed, by way of GMail (since any outgoing SMTP connection from my mail server would, of course, be ignored) what I expected to be the Usual Suspects:

Date: Sat, 21 Feb 2009 15:29:05 -0500
Subject: Could you explain why you’re blackholing my source IP address?
From: gabriel rosenkoetter
To: abuse@dreamhost.com, postmaster@dreamhost.com, webmaster@dreamhost.com

So far as I know, I have done nothing to attack you. My network is not open,
my mail server is secure with valid reverse DNS entries, and I’m not running
any webscraping. Could you please explain why you have blocked my access to
your customers’ web sites and provided me with difficult at best recourse to
contact you?

Dubious that anyone would see either of those (identical) missives, I reached out to some of my contacts in the Industry. By which I mean I asked whether anybody in the IRC channel where some of us sysadmin types hang out (no, that’s no longer EFNet #root for me any more; long story, politics, drama, and baggage) knew anyone at Dreamhost. One responded promptly in the affirmative and was very helpful in passing my query along. Here’s the thing about the lazyweb: you get better results with a self-selectedly intelligent audience, which, curiously, you rarely find on the world wide web proper.

And then I went out for the evening and straight to bed afterwards, only to rise to actual Results the next day! (Well, okay, fine I saw the reply on my iPhone while I was out, but I was in neither mood nor position to deal with it properly then.)

It appears (but I’m not certain: I put the same text in the web form and the email) that it was actually the web form (which I guess involved Sales, since nothing else seemed appropriate and they seemed like the folks who might care that someone couldn’t get to their customers’ web sites)—but maybe it was imajes’s influence? The world may never know!—that yielded the only unautomated response:

From: DreamHost Sales Team
Subject: Re: [REDACTED@gmail.com [REDACTED]] Could you explain why you’re blackholing my source IP address?
Date: Sat, 21 Feb 2009 18:09:29 -0800 (PST)
To: gabriel rosenkoetter

Thanks,
Phiya C

To which I, of course, promptly responded:

From: gabriel rosenkoetter < [REDACTED]@gmail.com>
Subject: Re: [[REDACTED]@gmail.com [REDACTED]] Could you explain why you’re blackholing my source IP address?
Date: Sun, 22 Feb 2009 12:39:01 -0500
To: [REDACTED]@dreamhost.com, [REDACTED]@gmail.com
Cc: DreamHost Sales Team

In many traceroute implementations, including FreeBSD’s (which this is), !X
means “communication administratively prohibited”, so I’m pretty sure that
there is an ACL on that pnap.net border gateway that’s dropping packets.
(The same thing happens with TCP packets.)

It took another 36 hours or so (which I am certainly in a position to understand; who knows, maybe an electrician decided to take one of their production DCs completely offline like one did to mine), but that then yielded this:

From: DreamHost Sales Team
Subject: Re: [[REDACTED]@gmail.com [REDACTED]] Could you explain why you’re blackholing my source IP addr
Date: Tue, 24 Feb 2009 16:28:52 -0800 (PST)
To: [REDACTED]@gmail.com

Thanks,
Brian S

I am just now, as in after this is posted, getting around to responding to ask whether they’ve got any change management documentation from when that block was put in place to document what I appeared to be doing because, as previously mentioned, I like to think I run an acceptably sound ship, and I’d like to know if I’m screwing up in some way so that I can fix it.

Overall, however, I am very, and unexpectedly, impressed with how well Dreamhost responded. Bravo!

Update: (2009-03-07 15:30 UTC-0500) Well, nothing’s perfect. They definitely didn’t understand what I meant when I asked for logs of any attack, mostly that I’m not one of their customers, but I do understand their privacy concerns:

From: DreamHost Sales Team
Subject: Re: [[REDACTED]@gmail.com [REDACTED]] Could you explain why you’re blackholing my source IP add
Date: Sat, 7 Mar 2009 06:10:12 -0800 (PST)

Thanks,
Javier R

Oh well.

1. Schedule maintenance on a PDU in the middle of a day, on a week dayWednesday.
2. When questioned over this obvious risk in the change management meeting by the Director over Windows systems, assure everyone that nothing could go wrong because the circuit is redundant.
3. Permit your electrical contractor to blow a fuse when performing this work at 11:30 US Eastern.
4. Take down all of the production HP-UX Oracle systems, most of the prod SQL Servers, all of the ESX servers in that site, several network core switches, one blade chassis, the OpenView server you assured us you were migrating off a year ago, so we took it off hardware support (which subsequently fails to boot because one of its PSUs has failed), and assorted other prod systems.
5. Rather than acting in your capacity over Operations, establishing a bridge call, and following the documentation your direct reports have created around order of startup after a site failure, permit chaos to run wild for three hours until cooler heads insist on at least a touch of Order.

Why, yes, I did have fuN at work today!

I didn’t really make much noise when I sucked this content out from Livejournal and put it up here (although maybe I’ll go over there and note that this exists, now that I’m comfortable with its stability), but now I’ve also finally gotten around to something I’ve meant to do for some time: /var/log/gr, wherein you will find links to various RSS feeds that I produce (including one from here).

First off, the reason to do this is that lots of places (notably, anything Brad Fitzpatrick touches…) now weight PTR records matching things like /(dsl|static)-[0-9]+-[0-9]+-[0-9]+-[0-9]+/ as indicating spam. Which, you know, I understand, but it’s a pain for people who maintain their own mail servers and have one of the less-than-ideal ISPs. When I moved into Philly, I was forced to switch from Speakeasy (who I liked very much, and do PTR records without any muss or fuss quickly) to Verizon Business DSL (you don’t get static IPs from Verizon without the business service), and I presumed that getting reverse DNS was going to be a hassle, so I put off doing the research.

After I got around to doing the research, I read various accounts describing pain and irritation, but found several that were successful of which Dean Gibson’s post to the postfix mailing list is the most complete I bothered to record in delicious:

http://archives.neohapsis.com/archives/postfix/2003-07/2967.html

That made made me think it wasn’t going to be a big deal, but I put it off some more (”about a year”, I think) because I was lazy and wasn’t seeing all that much lost mail. I’m still both of those things, but I was going through doing some setup for email at moonbar.net for some of the Apothecary folks (configuring SPF for the domain to forward allowability on through to _spf.google.com, so that they can just spoof the source address in GMail and not deal with my requirements around IMAPS and SMTP with SSL authentication), so I figured I’d clean this up too. It turns out it’s even easier now! Here’s what you do:

1. Email mmvbts@verizon.com the IP address domain mapping that you want.
2. Call 800-483-5325, choose option 3 for “Verizon enhanced products”, choose option 3 for “domain name services”.
3. Explain what you want and that you’ve sent the email (the account isn’t monitored, so you do have to call to tell them to look at it), verify your account information.
4. Wait 24 to 72 hours.

I’m currently in step 4, so I’ll update after I see how that goes…

Update: All set!

> server ns2.bellatlantic.net
Default server: ns2.bellatlantic.net
Address: 71.252.0.72#53
> 71.242.125.164
Server:         ns2.bellatlantic.net
Address:        71.252.0.72#53

164.125.242.71.in-addr.arpa     name = stow.eclipsed.net.

(Yes, they really do still have their primary external DNS in the bellatlantic.net domain.)

eAccelerator.

No, it's not perfect English (Oxford, American, whatever… Standard {Written,White} English), but it makes its point clearly and concisely. Like a blog, it's still conversational (in a way technical documentation “shouldn't” be), but that isn't getting in the way of my understanding the content.

Yes, eAccelerator is easier to describe than the sprawling blob of PHP, but the fact that you're documenting a sprawling blob is no excuse to start writing like you stopped paying attention in English class during fourth grade.

(I note that this maybe sounds like I'm saying, “Speak English!” I don't mean to be. I'd have a harder time processing properly-written Italian or German, and I wouldn't have a prayer in French, Russian, Japanese, and so forth, but I would be completely lost if I were reading Italian or German as poorly written as the English PHP documentation. I have a feeling that someone for whom English was not a first language would be totally stymied by the English PHP documentation. That's simply unacceptable by any sane consumer of the product.)