Loading…
27
Messed up the subnet mask on a VLAN config and took down half the office
Honestly, I was working remotely from a coffee shop in Denver and fat-fingered the subnet mask on a new VLAN, which created a routing loop that killed access for about 40 users for 20 minutes. Had to SSH in from my phone and roll it back while the help desk was blowing up my Slack. Anyone else accidentally nuke their network from a laptop in public?
2 comments
Log in to join the discussion
Log In2 Comments
evaperez4mo ago
Oh man, that is rough. I feel that pain right in my gut. I mean, there's nothing quite like the panic of realizing you just broke something for half the company while you're sitting there with a latte in your hand. Good on you for getting it fixed from your phone though, that takes some serious fast thinking. I've definitely had my share of "why did I think making changes from a sketchy WiFi was a good idea" moments. It's scary how one tiny number can just wreck everything, and then you have to deal with all the angry messages while your heart is pounding.
4
drew_west4mo ago
Spent 45 minutes on a train outside Chicago last year fixing a similar mess. I had the wrong default gateway set on a core switch and it created a broadcast storm that took down our entire third floor. Had to use a mobile hotspot and teamviewer into a jump box while the train kept losing signal in tunnels. Ended up keeping a cheap tablet with a wired USB keyboard in my bag specifically for these remote emergencies. Now I always double-check my subnet calculations twice before pasting anything into production, especially when I'm on public wifi.
3