Editorial: Symphony Of Destruction – Have we lost a network engineering culture?

by Brian Boyko
Editor, Network Performance Daily

I remember my first computer.  I remember deleting files to save space on it – files like “config.sys” and “autoexec.bat.” 

I remember my second computer.  I remember forcing a RAM chip in too hard and breaking off a piece of the slot. 

I remember my third computer.  I remember not grounding it properly. 

Around that time I started working in a mom-and-pop computer repair shop.  I remember accidentally reformatting the wrong hard drive. 

Luckily for me, I was generally better at computer repair than the one incident otherwise this would be a very long editorial. 

Does this story have a point?  No, but I’ll shoehorn one in anyway: What I’ve found is that as geeks we learn through trial and error.  The best of us become best at it by making mistakes and by learning from them.  Sure, we may have accidentally fried a few CPUs, or, in some cases, siblings, but the point is that the surest way to learn something is to do it, learning from mistakes.

One of the problems with rote memorization and studying for certification is that it is through our mistakes that we can discover strange new things – the phrase that heralds the greatest advances in science and engineering are not “Eureka!” but “Hmm… that’s funny….”

And with a very long rant, a poster on the Overclockers Australia forums talked about how things in IT are going from “bad to worse.” Reading the whole thing is time consuming, and there’s some salty language, but it’s worth it.  Here’s some highlights:

Why is it then, that over the last three years I’ve seen fewer and fewer people who call themselves sysadmins understand these things? Why is it that I’ve been surrounded by “IT professionals” from junior sysadmins to CTOs who don’t have a g*****n clue about one tenth of the above? Why is it that in three years I’ve met ONE person in professional IT who I would consider worthy of sitting down and having a conversation with?

Why is it that professional IT services today consist of service reps who tell you the things you are doing are untested, dangerous, unsupported, different, not usual, or a host of other words meaning they are scared s***less and unwilling to learn something new? Why is it that I spend my time building things people tell me for 6 months during build and test “will never work”, only to have them go into production and work ten times faster for one tenth the cost of the old system? Why is it that IT professionals today choose brand labels over intelligence, and post-justify it by hiding behind “board confidence” when providing a solid, working, profitable system is the best thing to boost confidence from the board?…

Many moons ago, I used to have a mentor. A man who quite frankly I considered genius level. I don’t throw around words like “genius” frequently. In my life I’ve met three people who would rightly qualify as geniuses. Only one I’ve had the pleasure to work with, and more importantly learn from. In the small amount of time I worked with the man my rate of learning tripled. He had the right amount of sage advice coupled with the sense to let you make your own mistakes from time to time.

This, I think is one of the biggest elephants in the room with regards to network engineering in general and network performance specifically.  There’s a reason we cherish “the old guys” in IT – these are the guys who were working on networks when it wasn’t a big deal – or at least, it wasn’t as big a deal as it is today – if they were broken.  In an environment where fault is expected and accounted for, you had a little bit more freedom to experiment.  Sometimes this leads to trouble.  Other times it leads to insight.  You repeat your insight and remember not to repeat your failures.

But the thing about enterprise networking is that the only people who own infrastructures large enough to support large networks are the companies who can absolutely not afford those large networks to fail or under-perform for any reason.  Going with the “safe” option instead of the one that offers the potential for learning is the only option that makes sense to the business.  But it is stunting IT.

I often wonder why computer security gets more coverage than computer networking.  After all, computer security doesn’t do anything, it just protects the stuff that does something.  Computer networking allows you to do amazing things.  Some of it, no doubt, is due to the “hacker mystique” – a fear of an active agent of destruction is more powerfully on the mind than planning how to eek out a 1% performance increase.

But there’s also the fact that security, rather than networking, is one of those fields where you are encouraged to try to get in there and break things.  This process of breaking things (and putting them back together) leads to more learning and more innovation… it’s a hell of a lot more fun and you learn more. 

Maybe what we need is for someone – I don’t know… academia?  The government?  Cisco?  – to develop a simulation of a working multinational Fortune 500 WAN, and just let students go hog wild on it, destroying and recreating it many times over, each time learning.