Greetings to you, empty void of the internet.

As I mentioned in my last blog post, I would be updating you on how my recent homelab project has come along. Thankfully, after about a week and a half of troubleshooting, I am finally done with that project and I can tell you one thing for sure: I will never purchase another HP server again.

Let me give you a quick rundown of what my last Friday night was supposed to entail:

• My good buddy Douglas Fritz was supposed to come over and eat Texas Roadhouse with my wife and I before we got to work on the project
• We were supposed to reroute some ethernet wires, re-end them, and put some keystone jacks on shorter runs to lengthen them
• Afterwards, we had planned to rip out my HP server and put in a PCIE cage to hold a Quadro P2000 I had purchased secondhand
  • This GPU was going to be used to passthrough to a virtual machine I had running on my hypervisor for media purposes
• Once we were done with the server, we were going to quickly re-rack it and start work on installing and configuring a new 1GBE switch I needed because I was running out of free ports for my hard-wired devices

What actually transpired though was a lot more headache than I had originally bargained for. The first of many disappointments came in the form of Pizza from Papa John’s when we failed to account for Texas Road House’s long wait times on a Friday night. To keep it short, by the end of the night we had managed to pull the server out, put the GPU in, and half reroute a of couple cables.

Around 9 PM, we managed to run into one of the biggest roadblocks that we would experience during this project: my server fans. Thanks to HP’s proprietary iLO firmware and “Sea of Sensors”, as they call it on their servers, my fans shot up to an unacceptable sound level that I could hear throughout my entire house. The server was trying to protect itself from overheating because the P2000 I just put in must have had an unrecognized firmware to my server’s lights-out management system. Because of this development, I had to scour the internet for solutions, finally landing on two possible options.

1. I could solder and wire all of my fans up to a microcontroller that could control all of my fans independent of each other (no thank-you)
2. Or, I could install a hacked-together modified firmware for my iLO card on the server that controls the fan speed

To me, the only logical choice was to figure out how to install this firmware and get my server back in the rack ASAP so I could send Doug on his way home (without him, I never could have lifted that behemoth). From the Reddit thread I quickly found, it seemed simple. I just needed to run a couple commands and I would have access to a whole suite of commands in my iLO card’s SSH interface to control fan speed based on the system’s sensors.

One big problem though was that I needed Ubuntu in order to run these commands to upload the modified firmware and my server was running Windows Server 2019. To interface with the iLO card, I would need to be running Ubuntu; however, I didn’t think I was ready to go full-Linux for my hypervisor quite yet. Normally just ripping out a drive and installing a Linux ISO onto it isn’t an issue for me in the slightest, but my SSD was plugged directly into my server’s RAID controller.

When booting into the RAID controller, I wasn’t able to see an option to just disable my Windows volume, so after quite a long time I tried to rip the drive from the server hoping the RAID card would be smart enough to re-recognize the volume when I put the drive back in.

No luck.

When the server was booting up, I kept getting an ominous message saying that I had two options:

1. Disable all logical volumes completely (which wouldn’t work because I was attempting to install Ubuntu on a separate HDD I had laying around)
2. Fail out the drive and kick the volume into recovery mode

Being too scaredy-cat (even with an on-site and off-site backup), I couldn’t for the life of me select number two. I tried selecting the first option a couple times hoping it would just temporarily disable the logical volume containing my Windows install, but the feature worked as advertised.

Eventually, I realized I could try installing both OSs side by side and change the boot order to the Ubuntu volume. I had always knew this was an option, but was trying to avoid it because of issues I’ve had in the past with GRUB wiping out Windows Boot Loader and making my Windows install unbootable.

Thankfully, after a lot of trial and error, this worked without borking my install of Server 2019 and I could finally run the commands to install the modified iLO which would allow me to control my fans. After a quick couple commands on Ubuntu, I was able to SSH into my server’s iLO and at last quiet the constant screeching noise I had been plagued with the last six hours.

We were able to stick my server back in the rack and I was able to send Doug back on his way so we could both hopefully sleep off this painful experience. The next day, I was able to finish up a lot of the wiring and get my new switch up and packet switching for us. What followed was a week of troubleshooting issues related to my media server. Long story short, I was able to get in touch with a Plex developer who straightened up all my issues eventually.

Of course, I didn’t include all of the problems I ran into with this project – that would take more time than you have. Some notable exceptions though are as follows:

• Not being able to transcode media files to under 1080p with Plex
• Switching to Jellyfin (and then back to Plex) for my media server
• Essentially ruining my reverse proxy install and needing to reconfigure it
• Not being able to install Ubuntu because of DNS issues
• Finding out not all of my wall sockets in my home are grounded (ouch!)

Sorry if this was a long read (trust me, it could have been MUCH longer), but my hope is to document the process of my projects with this blog – not the final outcomes. If you’re interested, I’ll have you know that the project did turn out to be a success and I was able to accomplish everything I wanted to with the help of my buddy Doug.

I don’t have any projects planned for the immediate future, but I have been thinking about getting an Ubiquiti access point for my home so I can segment my network traffic a little more. Currently all of my IoT devices sit on the same VLAN as all of my secure network traffic and something doesn’t sit right with me knowing Google, Roku, and others can see my networked devices. I think it’ll be time to cut them off soon!

-Bill

(P.S. I hope you enjoy this post’s picture of my little buddy Rudy)