The one thing I hate more than anything else is seeing people get the blame for something that they didn’t do because the people in charge are ignorant about technology. No, wait. Paper cuts. I hate paper cuts more than anything else. But that whole “travesty of justice” thing – very close #2.
This (possibly true, possibly not, definitely plausible) story from The Daily WTF had me ticked off. “Cam” – apparently a pseudonym – had set up a Web cam so he could prove to his bosses that he was working from home instead of just saying he was working from home. But on that day….
During a quick lunch break, Cam got a panicked call from his boss’s boss, Ron. “Cam, do you still have your webcam on?”
“Yeah, wh-”
“Turn it off. NOW,” he said in all caps over the phone.
“Uh, ok.” Cam flicked the switch on the webcam off. “So, why exactly is it so urgen-”
“Can’ttalknowbigproblems-” *click*…
See, it seems that there was a brief but major hiccup in a router somewhere between the bank’s data center and their T3 provider, causing a dramatic slowdown in outbound network performance, which rippled out into hundreds of branches and affecting thousands of online banking customers. In the troubleshooting process, the lead network engineer caught wind that Cam had been “streaming live video” over the network, and was going to tell! He complained loudly to Ron that Cam had caused the issues and lost some revenues for the bank in the process. Adding to this theory was the fact that the issue had apparently resolved itself close to the time that Cam turned off his webcam.
One week later, Cam is sitting with his boss Joel to discuss the issue. “Cam, I’m going to need you to sign this disciplinary action report before we file it with HR,” Joel said weakly.
Appealing to reason, Cam began, “Joel, you know exactly what happened. You know that all that was coming across the network was a static web page with a new image every so often. I never had more than five HTTP sessions at a time. It would take thousands, if not hundreds of thousands of simultaneous users accessing my web site at the same time to consume the bandwidth that it says I consumed on this report.”
“I know,” he said as his expression sank. Clearly, he’d fought for Cam and been overruled.
“Besides that,” Cam continued, “I’m hosting my site at my house. My upstream connection is capped at 360 kbps. There’s literally no physical way that anything I did from my house could even make a dent in our massive T3 lines, even if my upstream connection was 100% saturated!”
“I know,” Joel said as his face slipped into his hands. At this point, it dawned on Cam that he was lucky that all that was happening to him was a writeup. It sounded as though upper management would prefer to see him hanged. Still, it was absolutely unfair that he’d be made to take the fall.
“Furthermore,” Cam pressed, “what about our QoS policies? Surely internal users browsing external web sites have lower priority than-”
“I know,” Joel said again. “Look, I’ve fought them on this. You know I trust you, and that I know you wouldn’t ever — that you couldn’t ever — do something like this. I’m saying this as a friend; you’re better off just signing this. It’s not just you; management is pissed at me now, too. It’s not fair, but it’s how it is.”
In the end, there wasn’t much Cam could have done.
Of course, the network engineer who latched onto the “streaming video” theory should have gotten the blame for misdiagnosing the problem using the same kind of “If she weighs the same as a duck, she’s made out of wood and therefore a witch!” logic that can destroy the best laid plans of IT.
Maybe, if the moment the router started having huge performance problems, there was some sort of alert delivered to the network engineer – one detailing the problem, how bad it was, and where it was originating from, that might have helped. Someone might want to look into making something like that. Or – or, bear with me – if there was a way to look at the traffic patterns and Netflow data to see exactly how much bandwidth the Webcam was taking, providing exculpatory evidence for Cam, that might be nice. Someone should get on that.
In the meantime, I’ve just set up my own Network Performance Daily webcam to give you an idea of what my job is like. Hope you enjoy it!
No comments yet.