Suppose your IT organization is in the market for a new IT product (or service, but I’ll use the word “product” here to simplify the discussion). It could be a computer, network device or other hardware item, or it could be a software package or SaaS (software as a service). Regardless of what you’re looking for, you’ll probably go through a process that you’ve used over and over again for product selection. And there’s a good chance that your process is flawed. You won’t select the best product, and you’ll be burdened with having to support the wrong product for years to come.
It doesn’t have to be that way. In this article I’ll go through a list of common criteria for IT product selection, and I’ll give you my recommendations for change. Use this article to fine-tune your product selection process, and the result will be the selection of a product that’s a better fit for your true business needs.
The Most Common Product Selection Criteria
Here’s my list of the most common criteria for product selection, in descending sequence by the popularity I’ve seen. There’s no science here, and I haven’t taken a survey. These ratings are based on my own experience:
1. A checklist of “requirements”
This sounds like it ought to be a good way to select a product, but unfortunately the checklist is usually based on a wish list generated by the business users, with no distinction among what’s essential now, what’s important in the future, and what’s a nice-to-have. In simple pass/fail product selection, you narrow your list of product candidates by eliminating all candidates which have omissions on this checklist. The vendors know how this game is played, so most vendors have included capabilities in their feature lists which match the most common checklist items. Unfortunately, the vendor need to provide these checklist items is so great that most vendors stretch the truth. If their product really doesn’t provide a certain capability, they’ll figure out a narrow case in which the product sort-of fits the checklist description. Or they’ll provide a clumsy workaround to address the need, and then claim to have the capability.
My advice: Using a checklist of requirements is fine if, and only if, you:
- Limit the requirements to what you really need, plus what you think you’ll need in the future. Include nice-to-have features in a separate list if you want to, but don’t use the nice-to-have items in your product selection process to eliminate potential products.
- Push the vendors to show you exactly how each checklist item is provided by their product. If the explanation seems the least bit contrived, then talk to current customers of the vendor who are using the capability the vendor claims to have. Make absolutely sure that the vendor isn’t trying to satisfy the requirement in name only, with no commitment to provide the complete capability in an easily usable way.
2. The one your boss wants
This is the second most widely used selection criterion. It might even be a good way to pick a product, but you won’t know unless you know how your boss made the choice.
My advice: Don’t just take your boss’s word for it. Follow through with the other selection criteria and decide independently whether you agree with the boss’s recommendation/edict. If you end up agreeing, then fine; now you’ll have some reasons to support the decision. And if you disagree, then you’ll need all the ammunition you can get to make the case that you ought to go with an alternate vendor.
3. You trust the vendor
This is a popular selection criterion, but it isn’t usually stated as the explicit reason for the decision. Realistically, your job is on the line based on the product decision you make, and you’ll feel more comfortable going with a vendor you trust. That trust can come from the size of the vendor, the vendor’s financial stability, or a proven track record of vendor sales to your company in the past. It will often come from personal relationships that you’ve built with vendor representatives.
My advice: Make an effort to build a relationship with competing vendors. Find out whether you can trust them as well. If you only go with vendors you already have a trusting relationship with, then you’ll always go with the same vendors. That’s not necessarily a bad thing if your IT group is wildly successful with its current technology, if none of your competitors are doing any better than you, and if there are no technology breakthroughs in the future which might make your current technology obsolete. But given that all three of these things probably aren’t true, you owe it to yourself to look at other vendors and products.
4. The vendor or product is a “name brand.”
This is a variation on the “trust the vendor” criterion. If you hear a lot about a vendor or product in the press – even if the vendor is using its marketing skills to generate most of that publicity – then you’re more likely to trust the vendor or product. For my advice on this, see #3.
There’s a second aspect of name brands that ought to be considered. Vendors and products which are recognized as leaders often have a larger availability of skilled talent in the use of the product, and there’s likely to be more third-party training available.
My advice: think about how you will install, customize, support and train for the product if you buy it. If external resources are in short supply, what alternatives do you have? Factor their price (including any travel required to bring in support or training from outside your city) into your total cost decision.
5. Recent publicity for the product
This makes the name brand even more apparent. In fact, recent publicity for a product is often the impetus for looking at a product in the first place. Very often your CEO, COO, or other executive will hear about the tremendous advantage that a certain product is providing to another company, and will request the product for your own business.
My advice: The recent publicity tends to make the publicized product the front runner in an evaluation, but try to be objective. Look at the other criteria and decide whether the publicity is justified. Sometimes a less publicized product will be a better fit for your business.
6. You like the salesperson, or – perhaps worse from an objective evaluation viewpoint – your boss likes the salesperson.
This is once again a variation of the trust/name-brand/publicity criteria. It’s not really a good reason to buy a product, but it can end up being a reason why you don’t buy a competing product. See #3 and #2.
7. The cost of the product.
There are three variations of this criterion. Some people like to go with the least expensive product, feeling that they’re getting a bargain. Others like to go with the most expensive product, feeling that they’re paying extra for the best. And still others like to go with a product that is neither the most expensive nor the least expensive, like the idea of throwing out the highest and lowest score in judging certain Olympic events.
My advice: Don’t look at price; look at total cost and return instead. The total cost of a product is the price of the product plus all of the installation, customization, migration, training, adaptation, maintenance and support you’ll have to do or pay for. That is often quite different from the simple price. For example, a product that is less expensive might not have a key capability that your business needs, and you’ll have to pay for obtaining or developing a custom version of that capability. Make sure the work on that capability is factored into your total cost.
Similarly, look at the return to your business from buying and installing the product as a ratio to the total cost. Business people understand this as Return on Investment (ROI), but be careful to calculate ROI based on real expectations of costs and benefits. See Chapter 5 in my book to get a good feel for how to calculate a valid ROI.
8. The product fits your current or future architecture
Isn’t it sad that this criterion – possibly one of the most important criteria for determining the long-term impact of the product on your IT organization and budget – is relegated to number 8? In my book, Boiling the IT Frog, I say that, “The fewer [IT] products you have, the better off you’ll be, as long as you’ve chosen good products.” One of the primary contributors to excessive IT cost is the maintenance and support of a hodgepodge of divergent architectures and platforms due to a bunch of product decisions that have been made without taking architecture into account. And picking a product that doesn’t fit your current or future architecture will lead to inordinate amounts of time and money spent on integration, interfaces, training, wasted effort, and eventually migration to another product.
My advice: Know your current architecture, and decide how you want to change that architecture for the future (that’s part of your IT Strategy, which is an absolute necessity for a good IT organization). Don’t buy a product that conflicts with your architectural direction unless there’s an overwhelmingly compelling reason. To help justify an architecture-based product selection to your boss, estimate the long-term cost of a divergent platform. Don’t cave in on this issue – it’s important.
9. The reliability, resilience, security and auditability of the product
These are all factors which should be in your requirements (see #1) but are often omitted. The business tends to take them for granted and so everyone assumes that all products are equal in this area, but that’s not the case. New products (including even operating systems like Microsoft Vista) are often released before they’re mature enough for critical business use. They may work 99.9% of the time, but that .1% failure rate will wreak havoc on a business that requires a reliability of virtually 100%. And even if the product works most of the time, how much trouble is it to deal with the occasional failure? Can the product be properly supported in a resilient environment that automatically deals with hardware or network failure? Will the product increase the likelihood of security breaches or data leaks? And how easy is it to investigate transactions which appear to be invalid, inappropriate, or even illegal?
My advice: Include minimum standards for these factors in your requirements, and insist on hard evidence from the vendors that your standards will be met. And don’t just do this for new products – some products never mature.
10. Ease of Use
Here’s another factor that is often ignored in product evaluations. Our focus on requirements checklists (see #1) often neglects the way in which a particular requirement is met, even though the usability of the product will be critical in the long run. The problem with considering ease of use in product selection is that there are just too many different product capabilities in large products, and it’s impossible to investigate ease of use for every capability in every prospective product.
My advice: The answer to the overwhelming ease-of-use problem lies in the 80/20 rule, which states that 80% of the time using the system will be spent in 20% of the system’s capabilities. So look at what the users of the new product will do with the product, possibly by observing how time is spent in the current business process. Identify the processes and capabilities that will get the most use and attention, and focus on determining the ease of use for those capabilities. In extreme cases, you might even do benchmarks on two competing products (for example, measuring the time it takes to enter a batch of new orders using each product). At a minimum, you’ll assure your product’s future users that you’re paying attention to their needs. And you may prevent a potential disaster by choosing to stay away from a product that has a bad or inefficient user interface.
There are two kinds of flexibility that you ought to look for in any prospective products. First, you want the flexibility to be able to fit the product into your current environment. Vendors typically provide two ways to do this:
- Parameterization (this is my term – vendors may call it something else). This allows the adaptation of the software or hardware by changing settings, values or parameters. No actual change to the product is required, and no software code has to be written.
- Customization (again, this is my term). This will require actual software to be written, possibly as subroutines, extensions, or interfaces. Or for hardware, it will require add-on devices, fittings or fixtures.
A product which can be adapted to your needs using parameterization is preferable to a product which requires customization. The fundamental problem with customization is that you’re adding on to the product beyond the product itself. Then when the vendor modifies the product in future releases, your customization may no longer work correctly. In the long run, customization will be a big pain, and maintenance of customization will be a significant expense.
The second kind of flexibility you ought to consider is the flexibility associated with unknown future changes in your company, your business and your requirements. Businesses have a habit of evolving to meet the needs of their customers and their environment. This is a good thing, but it plays havoc with processes and systems that have become rigid in a quest for temporary optimization. The second kind of flexibility – which you may want to call adaptability – is extremely important for any industry. It protects your product investment against future needs.
My advice: make flexibility and adaptability important criteria, particularly as a tie-breaker between two more-than-adequate alternatives. Go with the product that supports your current needs through parameterization rather than customization, and that seems to be more adaptable to handle any business changes you might see on the horizon.
12. Do you really need the product at all?
Last, we have to wonder whether we’re asking the right question. Should the question be, “Which product do we want?” or should the real question be, “Do we need a product at all?” There’s a natural tendency among human beings to want to simplify their lives by employing the latest and greatest tools. This tendency leads business people to want an IT product because – they believe – it will simplify their lives. But we have to remember Larry Tesler’s Law of the Conservation of Complexity: “Every application must have an inherent amount of irreducible complexity. The only question is who will have to deal with it.” To apply the law in a slightly different context, every attempt to simplify the life of business people by adding new IT products will increase the complexity of life for the IT organization. It will also make the business more vulnerable to systemic errors that can lead to major business outages and vulnerabilities.
My advice: Try not to buy a product for emotional reasons. Look at the real total cost of the product (see #7) and the real total benefit from getting the product. Make sure that the difference (benefit minus cost) exceeds the risk of all of the issues possibly introduced by using such a product. If the difference doesn’t exceed the risk, then stay with what you have – don’t buy a new product.
Product selection is one of the most important aspects of a CIO’s job. This is particularly true for cornerstone products that form the basis of your IT architecture. The biggest mistake you can make is to turn the product selection process entirely over to the business users of a product. They know what bothers them about their current process, but they don’t understand the architectural environment in which the system must operate, they frequently don’t understand how their business processes fit into the overall corporate picture, and they often get distracted by product bells and whistles that won’t matter in the successful day-to-day business use of the product.
Focus on core requirements, work jointly with business users, get a product that fits your architecture, and buy from a stable vendor, and you’ll be rewarded with happy users and fewer headaches. Sure, some of the business complexity will be shifted to the IT organization, but that’s what they’re paying you for.