Recently a newsletter reader told me that his CEO asked the question, “Is our IT organization the best in the country?” The reader wanted to know how it’s possible to “benchmark yourself against other IT organizations so to be in a position to answer such a question.”
I think there are two answers to the reader’s question. The first answer is that the CEO needs some help in repositioning his thinking about IT. The CEO is viewing IT success as separate from the success of the overall business, and that view is a common warning sign of misalignment between IT and the business. IT is a part of the business – often a key part – and IT should be measured by its contributions to the business. If it’s a service business, then how does IT contribute to the processes used for delivery of the service to customers? How does IT improve those processes, providing more service at higher quality and lower cost? If it’s a product business, then how does IT improve the development and production of products and then facilitate the sales and distribution of those products? You get the idea.
My point is that it’s a trap to think about IT separately from the business. It reminds me of the old medical cliché, “The operation was a success but the patient died.” Clearly, a medical operation can’t really be a success if the patient dies, and similarly an IT organization can’t be successful if it isn’t contributing to the overall success of the business.
The tendency to measure process success independent of business success is not unique to IT. The United States has an award, the Malcolm Baldrige National Quality Award, that recognizes American businesses which meet specific criteria for process improvement. But what’s particularly interesting about the Baldrige award is that most of the business winners aren’t leaders among their business competitors. I’ve always had the view that companies who spend the time to qualify for the Baldrige award are spending their time on the wrong things – they’re focusing on their own processes and not on the way that those processes contribute to business success.
A second answer to the question is that there are two ways to measure an organization: you can measure it on what it’s doing or you can measure the organization on how it’s doing those things. Another way to say this is that you can measure whether the organization is doing the right things, or you can measure whether the organization is doing things right. Of course the ideal is that the organization is “doing right things right” – that the organization is doing the right things and it is doing them in the right way.
When I talk about an IT organization’s contribution to the business I’m mostly focused on what the organization is doing. Are the IT projects contributing to the bottom-line? Is the IT infrastructure resilient and secure enough to make the business customers happy? Where deficiencies in systems or infrastructure are noted, is the IT organization prioritizing its work to solve the problems and create the most benefit in the quickest possible way?
When you compare two IT organizations in two different companies – even in the same industry – you can’t easily compare what they’re doing, since the needs of the two businesses are likely to be substantially different and the two businesses are likely to be at two different points in their evolutionary path. It’s impossible to say whether one IT organization is better than another on this basis, except by asking the executives in the two companies whether they feel that the IT organizations are successful.
It’s like comparing the engines in two dissimilar race cars. Which engine is better? The one that makes the race car win the race, regardless of horsepower, compression ratio, or any other internal measure of the engine. Each engine has to be matched to the needs of its race car; you can’t measure the success of an engine without considering the needs of the car.
Inside-Out Measurement: Measuring How
I’ve been talking about outside-in measurements: measures of the success of something by looking at the effect it has on other more important things. There is some legitimacy in doing an inside-out measurement as well, but only as a secondary, less important measurement. For an engine, an inside-out measurement would be something like its horsepower-to-weight ratio, and these ratios can be compared between two engines. But again I want to emphasize that what’s truly important is whether the car wins the race – not its engine horsepower-to-weight ratio.
So with that caveat then yes, it’s possible to benchmark two IT organizations and measure their efficiency at doing things right – to measure how IT is doing things. You see this all the time in IT organizations because (a) the measurements are relatively easy to do, and (b) the IT people are very analytical and so they like to talk about these efficiency measurements. Common doing-things-right measurements include bugs per line of code, lines of code per person per week, up-time percentage of various hardware (A very deceiving number. Would you want to fly on a plane that has 99% up-time?), average phone wait time for support calls, average response time of a transaction (usually averaging across both peak and non-peak periods so that the average is nonsensical). I could go on and on, and many IT organizations have indeed gone on and on, with hundreds of these measurements. If you think about it, it’s almost certain that a creative analytical person can find some measurement in which his organization appears better than someone else. But the key question is, “Does the measurement have any correlation at all with business success and user satisfaction?”
I don’t mean to come across as totally negative on the idea of benchmarking and measurement. Some inside-out measurements truly are meaningful, and for those measurements, it’s a great idea to benchmark against other companies (or other organizations within your own company) to try to learn from the best practices of others. A measurement of unnecessary customer call-in hold time (for example, caused by the customer service rep waiting on a computer response) can be useful, and an improvement in the measurement directly correlates with business success. Similarly, the amount of time spent by an average salesperson filling out an order processing screen can be measured and improved through various processes, and the improvement can directly contribute to the productivity of the salespeople.
What you need to remember about inside-out measurements is that (a) they only matter if improvement in the measurement will lead to a tangible improvement in the business, and (b) it’s the processes that you’re comparing when you benchmark against another company – not the measurements themselves. It’s also important to keep the number of inside-out measurements small; too many measurements will prevent your employees from focusing on the process improvements that really matter to the business.
Do you have the best IT organization in the country? I don’t know, and you probably won’t know either. But the more important questions are,
- “Is my IT organization holding back the company, or is it contributing to the betterment of the business?” and
- “If my IT organization is contributing to the betterment of the business, then is it doing all that it can, and is it doing so in the most effective and efficient way possible?”
If you can answer that last question in the affirmative, then your IT organization is successful, and you can focus on the bigger question, “Do you have the best company in the country?” Ask your CEO to answer that question.