Flu shot and bizarre network issues

This last weekend I had drill and the medics gave us the seasonal-flu mist-vaccine. It's the one where they squirt gooey, inactive virus up your nose. I've had the vaccine before without any adverse side-effects. Yeah, not this time. I got the shot on Saturday and I was fine on Sunday. Not so on Monday. I woke up a few times in the middle of the night with a bit of a fever, but I figured that it would just go away. Yeah, didn't happen. On Monday morning I felt like I had been run over by a semi. My throat felt like I had swallowed bits of broken glass. Needless to say, I didn't go to work. I was bedridden most of the day, but towards the end I felt a little better. I tried to get a little work done but I wasn't too successful since I had a hard time concentrating.

By around seven or eight in the evening I was checking my email and I realized that Gmail wasn't picking up any mail from vivin.net. I thought that courier-imap might not be running, but it was. Gmail complained that the connection was timing out. Odd, because I hadn't made any changes at all recently other than an fsck after a power failure. I decided to try out the POP3 tester over here and I also got a timeout error. However, when I did a command-line test of the POP3 server like so:

openssl s_client -connect localhost:995

Everything seemed fine. Furthermore, this problem seemed to be limited to both my FreeBSD servers. My Linux box seemed to be running fine. I tried a bunch of things. I restarted both machines, restarted the network interfaces, made sure firewalls weren't running, reset my router... nothing seemed to work. Every time I tested the connection to the machines (using this awesome tool here), the connections would time out. What was even stranger was that a netstat showed connections to be in a SYN_RCVD state, which meant that the box had accepted the connection. It's just that the response wasn't getting out. Eventually I was so desperate that I decided to reinstall FreeBSD on one of the machines. I also posted to a BSD forum asking for help. I figured that next evening I could probably fix it after work.

Well, next morning I still felt terrible, but it wasn't so bad. My throat wasn't hurting as much and I didn't have that much of a headache so I was able to work from home. Oh, and the installation had completed. I set everything up quickly and gave it a whirl. Nope. Same problem. Yeah, I was getting pretty frustrated and annoyed at this point. I went so far as to contemplate wiping my beloved FreeBSD off both machines and putting Linux on there. I updated my post at the forum and I checked on it periodically throughout the day as I kept working. Finally, a kind soul popped in to help me out. He helped me troubleshoot using tcpdump and I was able to confirm that the machine was responding to the incoming connection, but it wasn't going out. Finally, he asked me to check the routing information. I'm not that well-versed with that so I didn't see the problem until he pointed it out. The routing information was all wrong!

See, I have two network interfaces on the box. One for my internal network, and one pointing to the outside world. The one pointing to the outside world is statically configured, whereas the one for my internal network picks up its configuration from my AirPort Extreme router. What was happening was that dhclient was overwriting the route information. So when the machine tried to respond back, it saw its default route as the internal router, and not my modem! I had to update my dhclient.conf and force the router address to the correct one:

backoff-cutoff 2;
initial-interval 1;
retry 10;
select-timeout 0;
timeout 30;

interface "vr0" {
   supersede routers 209.x.y.54;
   supersede host-name "enterprise";
   supersede domain-name "xxxx.xxx";
   request subnet-mask,
           domain-name-servers;

   require subnet-mask,
           domain-name-servers;
}

The important line here is the one that says supersede routers 209.x.y.54; (I've hidden the actual numbers from teh hax0rs). This tells the dhclient to override the router information that it gets from the DHCP server. Once I did this, everything started working again. My thanks to jggimi at DaemonForums!