My Neighbor's Wood Chipper Crashed My Server
It's a Tuesday afternoon, I'm sick with a cold, lying on the couch. Wanting to cover the sounds of my neighbor shredding through a year's worth of yard waste I pop on my headphones, pull up Jellyfin, pick a show, click play and... loading spinner spins and spins and spins. OK weird. Reload the interface and now Jellyfin is down. Such are the wages of self-hosting, I guess. I open a terminal and try to to ssh into my server, which is now... unreachable. Great.
With a groan I throw off the blankets, get up and walk over. Power light is on, but no network activity. Re-plug the network cable. Still nothing. OK, a reboot it is. I wait for the system to come back up. And wait. And wait. Finally it does, I'm pouring over dmesg and journalctl, great sick day activity, but there's nothing obvious. Except actually that long wait was a boot loop. 5 boot attempts before it came back up. Not great!
I plug in a Ventoy USB to rule out issues with the install or the boot drive and stop into the BIOS on the way only to find some extremely funky looking voltages. The 5 volt rail is reading at 4.7. The 3.3 volt rail is at 3 flat. OK, seems bad. Maybe the motherboard or power supply? Feeling demoralized and imagining the hit to my wallet I double check that the important services are swapped over to a secondary system and call it a night.
The next day I wake up ready to take this thing apart. Opening the case everything seems fine, if a little dusty. Bit by bit all the parts come out. Once everything is extricated I decide the power supply is the most likely culprit. I double check the pinout, set the jumper, plug it into wall power, and hit the switch. The fan spins up, good, but we already know it was getting this far before. Now for the fun part. I grab the multimeter, set the black probe on a ground and start testing each connection. All the voltages look good? I go through them again just to be sure.
I rebuild the core system on top of its motherboard box, as is traditional. Boot into the BIOS again, but this time our voltages are right on the money. OK? Just to be sure before I put this thing back in the case I bring up Ubuntu and run some stress tests while I have lunch, and we're now stable! So this whole time a cable just needed to be reseated? But what could have even caused that to shake loose? Suddenly the sound of my neighbor's wood chipper whirs in my memory and all becomes clear.
