I’ve always been pretty good at taking tests; I guess you could say I have a gift for “quizmanship.” But on a 1 – 10 scale, I would probably rate myself a 6 or 7 on the geek-o-meter. I’m not up there with some of the kids I once went to school with, who, when the teacher lapsed into a personal recollection, would immediately stick up their hand and ask whether this would be on the mid-term.
The nice thing about taking tests is that you’re dealing with a fixed environment in which there are hard and fast rules. You know the subject area to be covered by the test, and you have a general feel for the types of questions that will be asked. Reality is different: there are no rules, and you’re just as likely to need some skill that you’ve long ago forgotten as you are to be asked to take advantage of some recently acquired knowledge
Tests don’t measure reality – they measure a miniature subset of reality projected into a multiple-choice, short-answer, true-or-false environment. In our quest to be objective about the measurement of learning, we’ve zeroed in on the closest approximation to objectivity that our teachers can find. But it’s a poor reflection of the reality that we all have to live with after we graduate.
A couple of decades ago I taught a class to help people be more creative and to help teams work better together on creative solutions. The class included some exercises, and two of those exercises are applicable to my topic today. In one exercise we asked each person in the class to write down his or her best recollection of which letters are associated with each number on a phone keypad. This was in the days when phones were for making calls – not sending text or email messages – and so the exercise was difficult for most people.
After each person had recorded his or her own remembrance, we broke the group down into small teams, and each team was given the goal of developing a single “best answer.” Not surprisingly, when we compared the individual answers to the team answers, the teams generally were more likely to be correct than the average individual score. The team discussion helped people fill in the gaps in their knowledge. One person might remember one part of the keypad but be foggy on the rest, but the team was able to pull individual memories together into a perfect list of letters and numbers.
The second exercise was a little more complex. The exercise described a scenario in which you are part of small team of survivors from a plane that crashes in the desert 100 miles from the nearest civilization . The plane burns shortly after the crash, and you were all lucky to escape with just a few potentially useful items, including a compass, overcoats for each survivor, a cosmetic mirror, a gun, and a map of the area. Your task in the exercise is to rank the items in order of their usefulness to your situation. Again, we asked the participants to first individually rank the items, and then to get together into teams and agree on a team rank for the items.
The second exercise had a different result than the first exercise. In the first exercise, the best team score equaled the best individual score – this was to be expected since the problem was easier and a perfect score was more likely. But in the second exercise the team scores were generally better than the worst individual score but not as good as the best individual score. In other words, the teams gave an adequate performance but never as good as the most outstanding individual on the team.
3. Putting these two observations together
In a recent issue of ComputerWorld, Judy Estrin, author of Closing the Innovation Gap, made a comment with which I strongly agree. She said,
“It’s much easier to stifle innovation than it is to allow it to grow, and a natural inclination in the country and in the business world is to put in rigid processes, policies and metrics — things like Six Sigma or No Child Left Behind. Some of those things actually discourage the behaviors you need for innovation.”
I’ve written in the past about how bureaucracy cripples our ability to react to change. Here’s an excerpt from my October, 2004 newsletter:
When the word “bureaucracy” was first invented, it was supposed to be a good thing. Corporations were trying to cope with growth in situations where it was no longer possible to do enough hands-on management to ensure that the company implemented processes consistently across the enterprise. Bureaucracy was a standard set of processes and procedures which created that consistency even when implemented with different people, in different locations, and in different cultures. But the strength of bureaucracy—consistency—is also its biggest weakness; a bureaucratic environment can’t cope with changing conditions or with situations for which the rules weren’t designed.
Take another look at Estrin’s comment. Do you think of Six Sigma as bureaucracy? Many people don’t, and yet it fits the definition perfectly: Six Sigma accomplishes its goal of consistent high quality by using standard processes that minimize the deviation in the way things are built. The word bureaucracy has a negative connotation and the phrase Six Sigma generally has a positive connotation, but both have the same result: a rigid process.
What about No Child Left Behind? No Child Left Behind is a US government mandate to “improve” education by subjecting each child to a series of standardized tests. But the result of the law has been a change in the US education system to motivate teachers to “teach to the test” – to emphasize quizmanship rather than actual learning.
In my opinion, Estrin is absolutely correct in her criticism of both Six Sigma and No Child Left Behind. Both programs make overly simplistic assumptions about the real world. Six Sigma assumes that the right thing to do is to fine-tune your existing process or product to the point where there’s no deviation at all (sometimes valid, but less so in a constantly changing world). And No Child Left Behind assumes that a child who can do well on tests is best prepared for the real world.
Conclusion: The Cost of Oversimplification
How has the oversimplication philosophy of Six Sigma and No Child Left Behind affected your own IT processes? Are you measuring your software developers using crazy metrics like lines of code per week or – worse – by using “standardized tests” and certifications? Are you forcing your developers to use standardized processes that are long on documentation but short on creativity?
Think about that desert plane crash survival exercise. When you make heavy use of standardized processes and unrealistic measurement, you’ll get a result that is better than the worst individual but not nearly as good as the best that any one individual could offer. But what’s even more upsetting is that when your truly creative people are faced with overly rigid processes, they will get impatient and leave. You’ll end up with a bunch of mediocre and sub-par developers who somehow manage to collectively turn in an average performance.
You’ll end up with the IT equivalent of No Child Left Behind: everyone will be able to pass the test, but you’ll kill creativity, and you’ll have – at best – an average performing IT group. No programmer will be “left behind,” but neither will you have the outstanding developers who differentiate your business from the competition. Is that what you want? Is that what’s important to the business? I don’t think so.