Technology Analyst Firms Lack Rigor
Annual agendas followed by in-depth research should replace vague predictions

by Mark Smith | 2011-05-04 | Article ID: V11-10 | Article Type: VentanaView

Related Topics:

Printer friendly version
Email this article
Send feedback to editor


The technology analysis industry typically begins each year with a barrage of exuberant predictions. Most of these educated guesses fade as the months pass and are forgotten by the turning of the year unless a firm happened to hit one right and wants to tout its powers of prophecy. At Ventana Research we believe it is more valuable to publish for each of our practice areas an annual statement of direction that we call the research agenda. These research agendas state what we think is important enough to dedicate our time to, and they guide our analysts in the coming year. 

We make these research agendas available to clients and to the public. We challenge other firms to do likewise, and to in fact follow up their agendas with comprehensive research in each area they identify. Ventana Research does this, and using our benchmark research methodology we produce reports and indexes that provide practical advice to buyers in both lines of business and IT. Our research and the insights it yields make it possible for us to identify true best and worst practices, providing a research-based perspective on what to do and what to avoid. We invite all organizations seeking actionable information on the best uses of technology in business contexts to look to us for reliable results.


We are convinced it should be a standard practice in our business for each technology analysis firm to issue and then pursue an annual research agenda. Unfortunately, most firms just bring forward predictions based on their opinions and some inquiries of their own clients. Heterodox though it may be, we believe that clients have a right to expect that the analysts whose time they pay for will set out what their knowledge and experience leads them to conclude are the areas on which they should focus; how else are the clients to evaluate how well a given firm’s agenda matches their own objectives?

At Ventana Research we are guided in setting our research agenda by what our research and our client interactions tell us they need to advance their maturity and improve their competitiveness and performance. In practice, setting the agenda for each market segment requires comprehensive research across hundreds of organizations. We do this using the benchmark research methodology we have developed over the last decade that provides a framework for conducting, analyzing and presenting the research. We also conduct our research in both the business and IT sides of an organization. That’s because listening to and researching only one of these audiences can significantly skew the research findings and thus the guidance that results. Analyst firms that do not conduct primary research, depending instead only on client inquiries that in most cases come exclusively from IT, can develop biased views of markets. For example, some of the largest IT analyst firms have been slow in covering mobile technologies because, advising only the IT side, they do not see that the business side is driving the adoption of mobile technologies in the enterprise.

Nor is doing just buyer research sufficient. An analyst needs to understand the technology itself in order to evaluate its fit for the people and process needs of an organization. Such an understanding comes not from just receiving briefings from vendors, attending product demonstrations and reading product reviews; it is the result of actually engaging with the technology. Analyst firms that offer advice and guidance ought to have undertaken a detailed and informed assessment of different vendors and their products in order to understand them well enough to be able to match them to buyers’ needs. We do this using an approach that is modeled on a well-crafted request for proposal (RFP). It includes evaluation categories such as usage, capabilities and management as well as costs, benefits and an examination of other customers’ deployments.


Potential buyers seeking guidance need to understand that research methods vary among analyst firms. We suggest asking the following questions to determine the value of the services and advice you are receiving: How much hands-on experience does the analyst have in the business segment or technology that matters to you? In what ways does that analyst acquire his or her knowledge – from in-depth buyer research? Supplier research? Both? Neither? To what extent is his or her perspective derived from being a counselor to organizations rather than just from reading research and education that is already published? Does the advice you’re given speak directly to your current competencies and those of your organization, or does it just regurgitate advice drawn from previously reported adopters and implementations? We urge you to take all the steps necessary to ensure you can trust the information and advice you are utilizing for your business and IT organizations. After all, what’s at stake is the performance of your business. 


 Copyright © 2016 Ventana Research All Rights Reserved :: Privacy Statement :: Contact Us ::