A power outage at our main office made me rethink our entire network design

We had a full power cut at our downtown building last Tuesday afternoon, right in the middle of the workday. The backup generators for the servers kicked in, but our core network switch in the main closet didn't have a UPS. It just died. For about 45 minutes, our two other regional offices couldn't access any of the shared files or the main line-of-business app, even though their own power was fine. I was on the phone with our ISP, and the tech said, 'Your WAN is up, but your core is down. That's a single point of failure.' It hit me that we had built everything to rely on that one physical switch in one room. I've spent the last three days drawing up a plan to move to a more spread-out design, maybe with some failover at the other sites. Has anyone else had to rebuild their network layout after a similar wake-up call?

2 comments

2 Comments

chen.phoenix1d ago

Wait, no UPS on the core switch? That's wild.

victor_lane601d ago

Right? It's like they want the whole network to go down every time the lights flicker. Saving a few bucks on a UPS just costs way more in downtime later. I've seen places try this and it never ends well. Honestly, it feels like basic stuff at this point.