The Mythical 5%

http://www.artima.com/weblogs/viewpost.jsp?thread=221622

 

Computing Thoughts
The Mythical 5%
by Bruce Eckel
December 28, 2007

Summary
In November, I gave the commencement address for Neumont University, a school in Salt Lake City dedicated to teaching computer science where my coauthor lectures. This is that speech.

---

 

So here you are, about to be unleashed upon the world. This is a lot like taking someone and teaching them all the parts of English speech and then saying "go out and write stories," but it's probably all any school can do. And a more theoretical background wouldn't help; this is not to say theory is bad because you'll need some eventually but there's a recent movement that preaches "practice before theory" and I, apparently like Neumont, find that knowing how to do something is very helpful when trying to understand the theory behind it -- for me, the theory takes a lot longer to grasp and it's very useful to be able to be functional while my brain is catching up.

There's only so much you can learn in school, and it's only tenuously connected with what you will end up doing in your work. So I'm going to try to soften the blow a little, and arm you with a few insights that will help when you crash into the actual world of software development.

You've come from very structured learning. We even call it a science. It promises that there will be the same kind of structure and activities in the world. More importantly, you've come to expect a certain high level of success, similar to what you've experienced with your projects and assignments. But the world will not behave.

The statistics are sobering: 50-80% of programming projects fail. These numbers are so broad because people don't brag about their failures, so we have to guess. In any event, this makes the world sound pretty unreliable. Engineering gets better results, mostly because it has to. Bad software usually just annoys people but bad engineering can kill.

An even more fascinating metric is this: 5% of programmers are 20x more productive than the other 95%. If this were a science, like it claims, we could figure out how to get everyone to the same level.

Let's say that this follows the 80-20 rule. Roughly 80% of programmers don't read books, don't go to conferences, don't continue learning, don't do anything but what they covered in college. Maybe they've gotten a job in a big company where they can do the same thing over and over. The other 20% struggle with their profession: they read, try to learn things, listen to podcasts, go to user group meetings and sometimes a conference. 80% of this 20% are not very successful yet; they're still beginning, still trying. The other 20% of this 20% -- that's about 5% of the whole who are 20x more productive.

So how do you become one of these mythical 5%?

These people are not those who can remember all the moves and have fingers that fly over the keyboard erupting system commands. In my experience those in the 5% must struggle to get there, and struggle to stay there, and it's the process of continuous learning that makes the difference.

Because of what I do, I've met more than my share of these people. They read a lot, and are always ready to tackle a new concept if it looks worthwhile. I think if they do go to conferences they're very selective about it. Most of their time is spent being productive, figuring things out.

The big issue is knowing that you're going after that 20x productivity increase. Which means getting leverage on everything you do. Never just "bashing something out," but using the best tools, techniques, and ideas at your disposal. Always doing your best.

And deeper than that, understanding that the leverage point doesn't always come from the obvious thing, or from what we've been told, or the commonly-held belief about what works. Being able to analyze and understand a situation and discover the hinge points of a problem is essential; this takes a clear mind and detached perspective. For example, sometimes the choice of programming language makes a huge difference, but often, it's relatively unimportant. Regardless, people will still spend all their time on one decision while something else might actually have a far greater influence. Architectural decisions, for example.

So you must learn continuously and teach yourself new technologies, but it's not that simple. It's definitely good to learn more about programming, but you can't just learn more about programming. For example, just in the world of code, here are two of the biggest pain points:

1. Code is read much more than it is written. If people can't read your story, they can't improve it or fix it. Unreadable code has a real cost, and we call it "technical debt."
2. Code reviews are the most effective ways to find software defects, and yet we usually "don't have time for them."

Coming from the world of ones and zeroes we'd like to believe that things are deterministic, that we can provide a set of inputs and get the same outputs every time. I know because I believed this for a long time, and it took some hard knocks when I was a physics undergraduate to begin -- only to begin -- to wake me up. Many years later I was in a workshop and one of the other attendees told me what I had been learning all those years; she said "All models are wrong. Some are useful."

We are in a young business. Primitive, really -- we don't know much about what works, and we keep thinking we've found the silver bullet that solves all problems. As a result, we go through these multi-year boom and bust cycles as new ideas come in, take off, exceed their grasp, then run out of steam. But some ideas seem to have staying power. For example, a lot of the ideas in agile methodologies seem to be making some real impacts in productivity and quality. This is because they focus more on the issues of people working together and less on technologies.

A man I've learned much from, Gerald Weinberg, wrote his first couple of books on the technology of programming. Then he switched, and wrote or coauthored 50 more on the process of programming, and he is most famous for saying "no matter what they tell you, it's always a people problem."

Usually the things that make or break a project are process and people issues. The way that you work on a day-to-day basis. Who your architects are, who your managers are, and who you are working with on the programming team. How you communicate, and most importantly how you solve process and people problems when they come up. The fastest way to get stuck is to think that it's all about the technology and to believe that you can ram your way through the other things. Those other things are the most likely ones to stop you cold.

In my first jobs, I saw lots of managers making stupid decisions, and so, logically, I came to the conclusion that managers and management was stupid. It's a commonly held belief in our profession: if you're not smart enough to deal with the technology, you go into management. Over time I very slowly learned that the task of management wasn't stupid, it's just very, very hard. That's why all those stupid decisions are still being made; management is much harder than technology because it involves virtually no deterministic factors. It's all guesswork, so if you don't have good intuition you'll probably make stupid decisions. Napoleon wanted lucky generals rather than smart ones.

Here's an example: some companies have adopted a policy where at the end of some predetermined period each team evaluates everyone and drops the bottom 10% or 20%. In response to this policy, a smart manager who has a good team hires extra people who can be thrown overboard without damaging the team. I think I know someone to whom this happened at Novell. It's not a good policy; in fact it's abusive and eats away at company morale from within. But it's one of the things you probably didn't learn here, and yet the kind of thing you need to know, even if it seems to have nothing directly to do with programming.

Here's another example: People are going to ask you the shortest possible time it takes to accomplish a particular task. You'll do your best to guess what that is, and they'll assume you can actually do it. What you need to tell them for an estimate like this, and for all your estimates, is that there's a 0% probability that you will actually get it done in that period of time, that such a guess is only the beginning of the probability curve. Each guess needs to be accompanied by such a probability curve, so that all the probabilities combined produce a real curve indicating when the project might likely be done. You can learn more about this by reading a small book called Waltzing with Bears .

You need to pay attention to economics and business, both of which are far-from-exact sciences. Listen to books and lectures on tape while you commute. Understanding the underlying business issues may allow you to detect the fortunes of the company you're working for and take action early. When I first started working I looked askance at people who paid attention to business issues -- that was suit stuff, not real technology. But those people were the smart ones.

A great new source of information is podcasts. Anyone can do these so many of them are bad, but there are some real gems out there, podcasts that specifically cover topics in our profession. I learn a lot from these, and they help me stay current.

Here's another thing you probably didn't learn here. Both the world of business and the world of programming is legendary for flailing about with fads that promise to get things done better. The easy fads are patently ridiculous, or become so in short order. The harder ones to spot contain some reason or truth that prevents you from quickly dismissing them. Sometimes you need to pick out the good stuff and throw the rest away, and to do this you need to exercise critical thinking.

I saw a grocery bag that said "studies show children and teens who eat dinner with their families at least 5 times a week are 50% less likely to use alcohol." The bag's conclusion was to start eating dinner together and that will prevent alcoholism. Is eating dinner together the dominant factor? Or is there something else that causes both eating dinner together and reduced alcoholism?

Here are some more which many people have seriously believed:

• Companies don't have to make a profit anymore. It's the new economy.
• Real estate always goes up, even if salaries don't.
• Or even: A university must be a traditional campus and not an office building.

It's even harder when you come from the world of ones and zeros where we really, really want to believe that everything can be deterministic. It's harder than that when you understand that adding people into the mix and scaling up a system changes the dominant factors, while everyone around you still believes it should all be deterministic.

There's a book that uses studies to debunk beliefs about managing people and projects; it happens to be software-based but the thinking could be applied almost everywhere. This book is called Peopleware ; it's small and fun to read and I recommend it to almost everyone. Alas, it really isn't a book of answers, it just firmly destroys many closely-held ideas about how people work in business situations that involve programming. The best thing about it is that it reminds you how easy it is to take a wrong idea and build a bad world around it. It helps you ask questions.

So when you go into your new job with your head filled with technical knowledge from the last couple of years, and you add what I've told you today, you may be tempted to assume that everyone at the company has foolishly gotten themselves trapped with bad ideas. But there's one more very important maxim from Gerald Weinberg which doesn't really answer anything as much as gives you a way to understand what happens. He says: "Things are the way they are because they got that way ... one logical step at a time." It's the legendary frog in the saucepan. So from your fresh new perspective things might look ridiculous, but remember that each decision on the way was made by someone weighing the issues and making what seemed like the best choice at the time. This viewpoint doesn't solve the problem but it can make you more compassionate about the people who are stuck there.

You'll need to make a lot of mistakes in order to figure things out. So be humble, and keep asking questions.