Loading…

Debate: Was last week's outage my fault or the vendor's? | Enterprise Networking - Librear

Debate: Was last week's outage my fault or the vendor's?

We had a major VLAN issue last Tuesday that took down our finance department for almost 4 hours. I had just pushed a config change to a core switch, and suddenly half the network stopped talking to the other half. My gut says it was a bug in the firmware update we installed two days before, because the config was exactly the same as our test lab setup. But my boss thinks I forgot to save the running config before rebooting, which wiped some key settings. The vendor supports my side and is blaming their own code, but they won't put it in writing yet. I've been going over logs all weekend and I'm still not 100 percent sure what happened. Has anyone else had a similar finger pointing situation where both sides looked guilty, and how did you figure out the real cause?

2 comments

2 Comments

richard_ramirez4mo ago

Man that whole situation brings back memories of a time we had a SAN go down and the vendor swore up and down it was our cabling. Turned out one of their engineers pushed a bad patch to the management interface during a maintenance window we didn't authorize. It took us three weeks and a full packet capture to prove it because their logs conveniently had a gap during that exact time. Your boss is probably just covering his own tail - I've seen managers blame the config save forgetting thing more times than I can count. But here is the thing, if the config was a mirror of your lab setup and the test lab was running the same firmware, the bug theory makes way more sense than operator error. Check if your switch keeps timestamped config backups somewhere, that is usually the smoking gun. Our network team started snapshotting running configs before every change after that SAN debacle and it saved us twice already this year.

abbyl494mo ago

Check if your switch keeps those timestamped backups running configs before every change. Could be the smoking gun if it recorded what was actually running before you pushed the update.