IT vendors increasingly use reviews from analyst firms in their marketing - how should we interpret these?

Today, it’s harder than ever to tell how good something is, whether it’s a hotel, suit or software solution. When I look online for reviews all I see is noise.

To help find the guidance we need, let’s consider some relevant terminology.

Defining “best”

I remember being in a bar once when someone asked for the best burger they sold. “Our best burger” the waiter said. The next person said, “I want the best burger”. “Another of our best burgers” the waiter said.

“No” said the first person – “only one of us can have the best burger and I asked for it first”. “But we might not like what the chef thinks is best anyway” said the second.

Best is subjective, but it also suggests a unique level of quality.

Defining “leader”

Being a leader in your field is easier to define. They’re the people others want to follow because they’re superior for a reason that others value. They might have capabilities or assets we don’t have, or they might have a reputation for being dependable. Another important aspect of leaders is they’re usually a subset of a larger group.

Differentiating between leaders and the best

To help me differentiate between the two I think of the Olympics. The “leaderboard” shows the “leaders” who are the gold, silver, bronze, and the other leading competitors. However, only the gold medalist is the “best” – although there’s been a few surprises where two people managed to win gold in the same event so they now keep spare medals.

Differentiation in today’s IT marketplace

Today’s IT marketplaces are getting busier and busier. The digital and cloud revolutions mean vendors now release new products and updates every few months rather than every few years. It’s also allowed a new generation of vendors to compete against established names – and be disruptive competitors to them.

To help give an opinion on product quality in these fast paced and every changing IT marketplaces, I’ve noticed an increasing trend of analyst firm reviews becoming marketing materials. This isn’t exactly how they were intended to be used, so how should we interpret this new source of opinion?

Analyst reviews

Analyst firms are companies packed with people who use their experience to give an educated opinion on something. In IT, they primarily produce benchmarks and reviews for similar products and provide leadership with guidance on how to manage IT value.

You’re probably most familiar with Garner’s Magic Quadrants and Forrester’s Wave reports that have been reading material for IT leaders over many years. They help them validate whether their enterprise's choice of vendors is likely, in the analyst's eyes, to still deliver the best value for them. They also help organisations understand the most suitable options when they need to implement something entirely new to them.

For those not familiar with them, I’ve posted a link to a recent Gartner Magic Quadrant for x86 server virtualization here. Very briefly, you can see Gartner’s view is that Microsoft are better than Oracle, but not as good as VMware.

For the rest of us, they’re a good way to keep up to date on what the trends in a specific market place are. I recommend you read those freely available for the product areas you work with. As well as reading about the products you work with, it’s good to read about those you don’t, so they’re always an interesting read.

Jumping to conclusions about being a “leader”

From reading a Magic Quadrant, you can see being in its leader category means just that – being amongst that review’s other leading vendors – not necessarily being the highest rated in the review.

When a vendor (and most do it) tells you they’re considered a leader by an analyst firm it still doesn’t mean they’re the best. In fact, in mature market places you’d expect all of the main vendors to be leaders because they’ve had so much time to improve their products.

Instead, you should use this “badge of status” as a way of validating whether they might be a good choice for your organisation to consider.

Jumping to conclusions about “the best”

When reading the Magic Quadrant I linked to above, you might be very quick to assume VMware are the best for your organisation. The best at what though? The best at meeting Gartner’s criteria, which is probably different to yours.

Why? Because most of their opinion is based on a product or service’s capabilities and feedback from people already using it, not which is most likely to integrate into your existing IT environment.

Very few IT decisions are made today without considering the existing environment. Therefore, you should consider an analyst’s “best” status as a strong recommendation – and probably smile if you’re already using it!
If you’re not, find out why the analyst rated something else the “best” and if its a a capability you desperately need but didn’t think was possible.

Finally

Finally, I hope I've opened up a new source of free research for those of you new to reading analyst reviews and for those who have been reading them for a while hopefully my point of view made you feel more comfortable about yours!