Teeeeejirrrrr

This topic comes up for me not necessarily as a rant, but as I was going through some old notes before tossing them away, and this issue came up. How much damage can an organization do to themselves by hiring discount, unqualified programming resources?

The answer: A hell of a lot.

If you own a small software development shop, do yourself a huge favor and keep reading.

Stupidity cost us about $250,000 over a 9 month period. Another $250,000 was lost for us in other means.

A previous company that I worked at had a habit of hiring low-cost coding staff to handle production quality servers- Not just any production machines… but the very hardware which was responsible for 100% of the revenue being generated for said company. This makes no sense as their client side programmers were generally top-notch, and among the more creative and competent that I have had the pleasure to work with.

Management also had a habit of protecting these junior staff members voraciously- for various reasons, but mainly economic. A single server-side programmer could cost two or three times more in salary to maintain. Also, some in the management felt that this staff member was a ‘worthy project’ that could become a superstar programmer someday. I mean, it worked for Darko Milicic during his stint with the Pistons, didn’t it?

I calculated that my previous employer definitely cost themselves $250,000 in lost cashmoney over a 9 month period for the mishaps they had in retaining ONE very junior server-side engineering resource. This was absolutely staggering to me. Another $250,000 could easily be accounted for in other means. I’ll get to that in a moment.

The other sources for my numbers are the assumptions that:

1. Yes, this programmer was causing problems nearly EVERY SINGLE DAY.
2. Five client/server development resources were utilized, at $20/hr in wage costs each time a bug needed to be fixed.
3. Eight QA staff were utilized, at $15/hr in wage costs to confirm said bug fix.
4. This underqualified developer introduced one bug per day, which took an average of six hours per bug to fix.
5. This individual worked for a period of 9 months, before finally being axed for causing a systemwide outage of all revenue generating products for a significant part of a business day.
6. Not entirely relevant, but said individual produced on average 15 lines of code per day, with the assumption of 21 working days per month. Vacation time is included in this count because this individual only really took one, and worked some weekends as well – so it all balances out.

The numbers I am leaving out are:

1. Costs to management salaries. The reason for this is twofold- one is that I don’t know how many staff members were involved in back-room decisionmaking each time we suffered downtime events. This ranges from one to four. Who knows…
2. Costs to my productivity. I was a systems admin, so it was my job to support the engineering staff. I had the unique experience of being able to read/understand code, but full well knew that I could not hold a candle to the proper engineers. In a nutshell, I could talk shop with them, and help diagnose problems. I was a bit faster than a plain old sysadmin at doing this. I was a good cheerleader at times. That’s about it.
3. Costs of contractors: During most of this 9 month period, we had consulting/contracting staff on board who had to shift gears to help fix problems on a near-regular basis. I simply did not accurately keep track of how often they had to drop whatever they were doing and scramble to help us.
4. The damage caused by spaghetti code. This is utterly incalculable. Each time new server-side programming staff were added they faced an absolute nightmare of a learning curve.
5. The amount of time lost by production server uptime – According to my notes we were down (and by that I mean not being able to process subscription material) for a period of nine total business days during that 9 month period. To this day I still don’t exactly know how many dollars per hour in transactions were being handled by those servers, so I can’t guess there. I won’t even bother to put a price tag on it, but it is the elephant in the room.

If I really do have to hazard a guess at the total cost of damage over 9 months, then I’ll say that stupidity cost us about USD $500,000 total. There is also the untold damage done to the reputation of the company, as subscribers probably grew tiresome of outages and took their business somewhere else.

How much damage can one incompetent programmer do to an organization? A hell of a lot.

In the end it isn’t my intention to throw anyone under the bus for the mistakes that were made. Guilty parties all know who they are, and to some degree or another, they all paid for it already, so there’s no need to flog that dead horse all over again.

The underqualified individual who boasted of general programming expertise which he did not have faced the prospect of never finding work in a proper engineering environment again. Management who felt they were saving money by having this resource around because of cost-effectiveness are facing the prospect of their entire company going down the tubes because the server-side codebase is nearly unmaintainable, among other things. Those of us who stuck around trying to fix the situation paid for it with a few sleepless nights, and the agony of ‘Heisenbug’ code which was nigh near unpossible to debug or maintain effectively.

I learned many things myself. I used to never document very well. Because of the insanity I was forced to deal with, I feel I have come a long way in that regard. I enjoy documenting now. It’s saved my ass a number of times since then.

Over a year after the staff member was let go, the server programmers were still fixing bugs.

Also, a young and promising programmer who was forced to help debug and manage the fix processes on a regular basis learned a hell of a lot from these failures, and it will serve him tremendously in his career as time goes on. This is the type of education you just can’t get in college. Over a year after the staff member was let go, he was still fixing bugs. The thing is, he’s already bailed on this company and isn’t looking back. If there was someone worth focusing on and making into your future team nucleus… this guy was the one.

What’s done is done, and hopefully dear reader, if you’re in management or if you own a small company, you won’t make the same mistakes that they did. I implore you to hire competent people. Don’t cheap out in that regard. It will cost you a whole lot more than you think.