Walk Away

One of the rules of troubleshooting is never change more than one thing at a time. Given that I have effectively become a professional troubleshooter as a sysadmin, you’d think that I would be capable of remembering this, turns out, not so much.

After spending the better part of 3 months acquiring, configuring, reconfiguring, and using my test lab ESXi machine, I decided it needs one last bit of reconfiguring. Since the purpose of this is to have a platform for testing exploits, it is a good idea to create a DMZ network to wall the virtual machines off from the rest of my home LAN. “This should be easy”, I told myself. Add a NIC to the router(an old Dell running PFSense) and one to the ESXi host(a less old Dell), connect the two and tell PFSense what to do with traffic.

Turns out it really is just that easy. Once the link is active in PFSense, you just add the interface, rename from OPT1 to DMZ just to clean it up, and set the IP. Next, set a couple of simple firewall rules to allow any traffic from the DMZ interface to anywhere that is NOT the LAN interface, and any traffic from the internet to the DMZ interface. Then just turn on a DHCP server, and away you go.

Away I go, almost. The link is up and physically active, blinky lights and all, but no DHCP. “How did you check this?” Good question, glad you asked. In the configuration of the ESXi host, there’s a network adapters section. Looking at this, the LAN interface showed the IP range that I had configured on the LAN interface DHCP server. I *assumed* the same thing would happen when I connected the DMZ link. “Didn’t you try to verify another way?” Yes, and here’s where I totally dropped the ball. I tried rebooting the router and the ESXi host, nothing changed, I tried reconfiguring the ESXi connection, I tried reconfiguring the DMZ interface on the router, nothing changed. I added the interface to a vSwitch, connected only that vSwitch to a VM, and tried to force its NIC to update, even rebooted the VM. “Didn’t you say you were a sysadmin? You couldn’t figure out networking?” I was in a hurry, so I logged into a VM I had never used before, thinking it would be just as good as another. I was wrong.

In frustration, and knowing that I was already confused by something simple, I stopped, and came back the next night. For good measure, I rebooted both machines. I logged into a different VM, Backtrack. I’m comfortable with the OS at a commandline and GUI level. My assumption this time was, “it’s another day, before you change anything, just give it a shot”. TA-DA! Now it works. Connected immediately, could ping the gateway(DMZ interface) IP, could ping google.com, distrowatch.org, you name it. Internet connection live.

So I changed configuration and tested with something I didn’t fully understand. This time it didn’t really cost me anything, because getting that interface working was the goal of the night. But it did serve as a reminder not to get cocky. I’m fairly comfortable troubleshooting simple networking problems, provided I’m using tools I am comfortable with. I’m also thankful it only took me 24 hours to find the solution.