Wine Intelligence admits Bias, Ulterior Motives in “Wine Blogger Distrust” Release
Consumers wary of blogger recommendations, according to research
Independent bloggers are one of the least trusted wine information sources in the UK, USA and France, according to research published today, despite the growing importance of the Internet as a source of information about wine.
In a post yesterday on their own site they begin to backpedal:
Arguably we took the sensationalist approach…partly because we wanted to generate some debate (and sell some reports). – Bloggers Bite Back, WineIntelligence.com (Feb. 4, 2011)
Wow, really? You mean we don’t need to shut down our blogs and get a gig for peanuts where someone else can dictate what we write? Alder Yarrow, who blogs from San Francisco on Vinography.com hits the nail on the head posting, “Quite a clever tactic to publish a study finding about wine bloggers that would likely prompt a lot of them to write about it, no?”
If a PR firm promoting social media pens a release on the same survey, it might read: “Around 20% of consumers say they trust recommendations from an independent wine blogger, and the number is growing.” It leads me to my favorite phrase from my least favorite MBA class, “There are lies, there are damned lies, and there are statistics.”
In delving a bit deeper into the Wine Intelligence report methodology, I find something a bit interesting, especially when it comes to the U.S. sampling,
1. Study Restricted to parts of the “Northeast” United States
Out of the estimated 75 million “regular” wine drinkers in the United States, Wine Intelligence sampled 1,255. Of that miniscule (.0017%) sample size of “regular still wine drinkers” (sparkling doesn’t count) Wine Intelligence says respondents come from only 10 states in the Northeast. But the Wine Intelligence folks are a bit dodgy about what constitutes a “state.” To be more specific, their sampling really came from only seven states (Connecticut, Massachusetts, Rhode Island, New Hampshire, New Jersey, Virginia and Maryland, with an additional three cities (New York City, Chicago and Washington, DC) Wine Intelligence fails to survey wine drinkers from the two most populous States in the nation (California and Texas).
They also ignore the entire West Coast region (California, Washington and Oregon) where the majority of American wine is produced. California is first with 89% and Washington is third with 3.34% following slightly behind New York at 3.69%. Oregon (Dept of the Treasury TTB Report for 2009)
It’s a bit like doing a report on the Oil Industry and ignoring the entire State of Texas.
While our readership is probably not your “typical still wine drinker,” we note the following AWB readership statistics in comparison. If you look at our readership over the past year, you get the following breakdown.
We can break our statistics down to major U.S. cities, and then group the cities into states. Nearly 78 percent of our readership is from the U.S., and the rest is from another 152 countries and territories all over the globe, including France, Italy, Spain, Portugal, Greece, Germany, India, Australia, Japan, China, Thailand, Argentina, Chile, Romania, Croatia, The Netherlands, Sweden, Switzerland, Brazil, Egypt, Israel, Iran, the United Arab Emirates, Poland, Denmark, Mexico, Russia, Poland, Turkey, South Africa, Scotland, Peru, Austria, Hungary, New Zealand, Iceland even the Isle of Man. But the important thing to note that the region on which the entire Wine Intelligence U.S. sample size is based, makes up only about 7 percentage of our U.S. readership. That leaves another 93% around the country whose opinions, according to the survey, apparently wouldn’t count.
2. Internet Savvy Respondents Answers May have been eliminated.
According to the methodology powerpoint presentation, “Invalid respondents (those who sped through the survey or gave inconsistent answers to selected questions) were removed before analysis.” That is, those who could complete the survey quickly, or perhaps had ADD from spending too much time solely reading on the internet, were invalidated.
In other words, people who are really adept at using websites or social media, may have been eliminated from a survey about the influence of social media.
3. Survey Quotas Not in Line with U.S. Population
According to the methodology,”The survey is quota-based; quotas are defined in terms of age, gender and region.” Now, just the word “quotas” disturbs a tea drinker or two. But note that in neither gender nor age, do the survey respondents match up with the overall population distribution of the U.S.
I don’t know what sort of “quotas” they use, and I don’t have the $2,150 to find out. But my guess is there is an inherent bias in their statistical sampling toward upwardly mobile white males between the ages of 25-34 with a significant amount of disposable income who spend more time in urban wine shops and wine bars than they do reading wine blogs on their laptop or Blackberry.
4. List of Wine Blogs Used in Survey Not Representative
Wine Intelligence provides 35 websites for its “internet and social media” usage question, stating:
Which of the following websites/forums/blogs do you currently use to search for, talk and learn about wine? Please select all that apply.
Of those 35 available for selection, exactly 10 appear in a search of the Top 200 Wine Websites based on monthly traffic:
Now you might wonder exactly what the Top Blogs might be. There is various methodology in ranking blogs, a bit like the BCS when a 6-5 SEC Team with a weak schedule can beat out a 10-1 WAC Team for a spot in the National Championship. However, I like to look at viewership, as they do with television ratings. So, these are ranked by average monthly traffic. Let’s just pull the Top 35. I pulled these from Cellarer.com, and removed the magazine and newspaper websites, so we could concentrate on actual blogs, since that’s what the folks at Wine Intelligence cited in their release. You’ll note there are a few more than 35, because several were tied in numbers of unique visitors at 9,422 per month. This may be the result of some sort of projected algorithm, because some of the self-professed more popular sites hide their actual traffic numbers from Mr. Google, and other easy-to-access analytics sites. We don’t know why they do, but we’re sure they have a good reason.
Social Media vs. Blogs
In releasing their findings the Wine Intelligence folks lumps together independent wine bloggers, Facebook and twitter.
To summarize our position: we asked our normal sample of regular wine drinkers in three markets (USA, UK, France) a few questions about their internet and social media use. It turns out that many of them use the internet now to find out more information on wine, and a smaller proportion of them use more “Web 2.0” stuff – Facebook, blogs, and other interactive message-posting sites.
Really? That’s a bit like grouping an opt-in targeted newsletter with non-solicited direct mail and door hangers. Sure they all use lists. Sure they each provide information. But I’m not placing the same value on the information I get from a random piece of mail to “Resident” or a flyer someone sticks on my door to a newsletter to which I actually subscribe. And I’d be much more likely to listen to the advice of trusted friend than some guy who just happens to live in the neighborhood. But the neighbor guy will certainly beat the unsolicited chatter of junk mail (or twitter or Facebook.) People read blogs for advice because they do trust them.
Wine Blogs as Word-of-Mouth Marketing
Sure there are some wine bloggers actually trying to make a living at the gig. They’re either displaced print journalists, or those who dream of fame and fortune in the world of wine. Some sell advertising. Some refuse samples. And some write positive reviews of mediocre samples just so they can get free wine. But many of us are not trying to compete with Decanter, Wine Enthusiast, The New York Times, San Francisco Chronicle or Wine Spectator. We’re not trying to make a living off of selling something to our readers, be it wine, wine accessories, gourmet food pairings, wine travel, or the winery clients who are paying for an expert (Soi-disant).
And here is where the surveys and parced responses go off the rails.
Comparing the wine merchant, supermarket newsletter or major wine publication to an independent blogger is juggling apples and oranges. Note the word independent. What we tell you to buy does not affect our bottom line. We’re simply another source of information. I might even argue that we are perhaps more trustworthy because we don’t have a “dog in the hunt.” The people of Wine Intelligence are asking you to fork over $2,150 USD for an analysis of data inferred from a slanted survey. A wine merchant is asking you to purchase the wines they recommend. A magazine features stories and reviews of wines that appear on full-page advertisements within the same publication. And yet those of us who write about our passion are to be trusted less?
Wine Intelligence admits that its press release was written to stir up controversy and to sell reports. They claim to be interested in feedback on their methodology.
The other main criticism is that we didn’t ask the right questions. Clearly this is an area that we can debate- in fact we would invite critiques of our questionnaire if anyone is interested in making them – since we need to develop an effective assessment system for this medium in comparison with more traditional sources of wine information. I look forward to hearing your views.
But you might find their sincerity to be just a bit far-fetched, given the fact that the comment section under said posting inviting critique, is closed.
Ours is open – comment away!
***UPDATE: On Friday, February 4 when it first appeared on Wine Intelligence’s website, the Comment Section under the post Bloggers Bite Back inviting feedback was closed. Since the posting of this story and its appearance on a number of social media and on-line publications, the comment section is now OPEN.