Wine Intelligence admits Bias, Ulterior Motives in “Wine Blogger Distrust” Release

By
[ 43 ] Comments
Share




Last week Wine Intelligence, the self-proclaimed “leading research-led strategy consultancy serving the global wine industry,” released the following:

Consumers wary of blogger recommendations, according to research

Independent bloggers are one of the least trusted wine information sources in the UK, USA and France, according to research published today, despite the growing importance of the Internet as a source of information about wine.

In a post yesterday on their own site they begin to backpedal:

Arguably we took the sensationalist approach…partly because we wanted to generate some debate (and sell some reports). – Bloggers Bite Back, WineIntelligence.com (Feb. 4, 2011)

Wow, really? You mean we don’t need to shut down our blogs and get a gig for peanuts where someone else can dictate what we write? Alder Yarrow, who blogs from San Francisco on Vinography.com hits the nail on the head posting, “Quite a clever tactic to publish a study finding about wine bloggers that would likely prompt a lot of them to write about it, no?”

If a PR firm promoting social media pens a release on the same survey, it might read: “Around 20% of consumers say they trust recommendations from an independent wine blogger, and the number is growing.” It leads me to my favorite phrase from my least favorite MBA class, “There are lies, there are damned lies, and there are statistics.”

A survey on U.S. wine information consumption that ignores the state of California is a bit like doing a report on the Oil Industry and ignoring the entire State of Texas.
What is inherent, yet often missing when survey results are announced is the sample size and make-up of that sample. Or perhaps it’s included but buried lower in the release. “Out of the 100 folks whose names we culled from Decanter subscriptions and the local supermarket customer loyalty card lists, 4 out of 5 say they get their information from places other than the internet.”

In delving a bit deeper into the Wine Intelligence report methodology, I find something a bit interesting, especially when it comes to the U.S. sampling,

1. Study Restricted to parts of the “Northeast” United States

Out of the estimated 75 million “regular” wine drinkers in the United States, Wine Intelligence sampled 1,255. Of that miniscule (.0017%) sample size of “regular still wine drinkers” (sparkling doesn’t count) Wine Intelligence says respondents come from only 10 states in the Northeast. But the Wine Intelligence folks are a bit dodgy about what constitutes a “state.” To be more specific, their sampling really came from only seven states (Connecticut, Massachusetts, Rhode Island, New Hampshire, New Jersey, Virginia and Maryland, with an additional three cities (New York City, Chicago and Washington, DC) Wine Intelligence fails to survey wine drinkers from the two most populous States in the nation (California and Texas).

They also ignore the entire West Coast region (California, Washington and Oregon) where the majority of American wine is produced. California is first with 89% and Washington is third with 3.34% following slightly behind New York at 3.69%. Oregon (Dept of the Treasury TTB Report for 2009)

It’s a bit like doing a report on the Oil Industry and ignoring the entire State of Texas.

While our readership is probably not your “typical still wine drinker,” we note the following AWB readership statistics in comparison. If you look at our readership over the past year, you get the following breakdown.

We can break our statistics down to major U.S. cities, and then group the cities into states. Nearly 78 percent of our readership is from the U.S., and the rest is from another 152 countries and territories all over the globe, including France, Italy, Spain, Portugal, Greece, Germany, India, Australia, Japan, China, Thailand, Argentina, Chile, Romania, Croatia, The Netherlands, Sweden, Switzerland, Brazil, Egypt, Israel, Iran, the United Arab Emirates, Poland, Denmark, Mexico, Russia, Poland, Turkey, South Africa, Scotland, Peru, Austria, Hungary, New Zealand, Iceland even the Isle of Man. But the important thing to note that the region on which the entire Wine Intelligence U.S. sample size is based, makes up only about 7 percentage of our U.S. readership. That leaves another 93% around the country whose opinions, according to the survey, apparently wouldn’t count.

2. Internet Savvy Respondents Answers May have been eliminated.

According to the methodology powerpoint presentation, “Invalid respondents (those who sped through the survey or gave inconsistent answers to selected questions) were removed before analysis.” That is, those who could complete the survey quickly, or perhaps had ADD from spending too much time solely reading on the internet, were invalidated.

In other words, people who are really adept at using websites or social media, may have been eliminated from a survey about the influence of social media.

3. Survey Quotas Not in Line with U.S. Population

According to the methodology,”The survey is quota-based; quotas are defined in terms of age, gender and region.” Now, just the word “quotas” disturbs a tea drinker or two. But note that in neither gender nor age, do the survey respondents match up with the overall population distribution of the U.S.

I don’t know what sort of “quotas” they use, and I don’t have the $2,150 to find out.  But my guess is there is an inherent bias in their statistical sampling toward upwardly mobile white males between the ages of 25-34 with a significant amount of disposable income who spend more time in urban wine shops and wine bars than they do reading wine blogs on their laptop or Blackberry.

4. List of Wine Blogs Used in Survey Not Representative

Wine Intelligence provides 35 websites for its “internet and social media” usage question, stating:

Which of the following websites/forums/blogs do you currently use to search for, talk and learn about wine? Please select all that apply.

Of those 35 available for selection, exactly 10 appear in a search of the Top 200 Wine Websites based on monthly traffic:

Now you might wonder exactly what the Top Blogs might be. There is various methodology in ranking blogs, a bit like the BCS when a 6-5 SEC Team with a weak schedule can beat out a 10-1 WAC Team for a spot in the National Championship. However, I like to look at viewership, as they do with television ratings. So, these are ranked by average monthly traffic. Let’s just pull the Top 35. I pulled these from Cellarer.com, and removed the magazine and newspaper websites, so we could concentrate on actual blogs, since that’s what the folks at Wine Intelligence cited in their release. You’ll note there are a few more than 35, because several were tied in numbers of unique visitors at 9,422 per month. This may be the result of some sort of projected algorithm, because some of the self-professed more popular sites hide their actual traffic numbers from Mr. Google, and other easy-to-access analytics sites. We don’t know why they do, but we’re sure they have a good reason.


Social Media vs. Blogs

In releasing their findings the Wine Intelligence folks lumps together independent wine bloggers, Facebook and twitter.

To summarize our position: we asked our normal sample of regular wine drinkers in three markets (USA, UK, France) a few questions about their internet and social media use. It turns out that many of them use the internet now to find out more information on wine, and a smaller proportion of them use more “Web 2.0” stuff – Facebook, blogs, and other interactive message-posting sites.

Really? That’s a bit like grouping an opt-in targeted newsletter with non-solicited direct mail and door hangers. Sure they all use lists. Sure they each provide information. But I’m not placing the same value on the information I get from a random piece of mail to “Resident” or a flyer someone sticks on my door to a newsletter to which I actually subscribe. And I’d be much more likely to listen to the advice of trusted friend than some guy who just happens to live in the neighborhood. But the neighbor guy will certainly beat the unsolicited chatter of junk mail (or twitter or Facebook.) People read blogs for advice because they do trust them.

Wine Blogs as Word-of-Mouth Marketing

Sure there are some wine bloggers actually trying to make a living at the gig. They’re either displaced print journalists, or those who dream of fame and fortune in the world of wine. Some sell advertising. Some refuse samples. And some write positive reviews of mediocre samples just so they can get free wine. But many of us are not trying to compete with Decanter, Wine Enthusiast, The New York Times, San Francisco Chronicle or Wine Spectator. We’re not trying to make a living off of selling something to our readers, be it wine, wine accessories, gourmet food pairings, wine travel, or the winery clients who are paying for an expert (Soi-disant).

And here is where the surveys and parced responses go off the rails.

Comparing the wine merchant, supermarket newsletter or major wine publication to an independent blogger is juggling apples and oranges. Note the word independent. What we tell you to buy does not affect our bottom line. We’re simply another source of information. I might even argue that we are perhaps more trustworthy because we don’t have a “dog in the hunt.” The people of Wine Intelligence are asking you to fork over $2,150 USD for an analysis of data inferred from a slanted survey. A wine merchant is asking you to purchase the wines they recommend. A magazine features stories and reviews of wines that appear on full-page advertisements within the same publication. And yet those of us who write about our passion are to be trusted less?

Wine Intelligence admits that its press release was written to stir up controversy and to sell reports. They claim to be interested in feedback on their methodology.

The other main criticism is that we didn’t ask the right questions. Clearly this is an area that we can debate- in fact we would invite critiques of our questionnaire if anyone is interested in making them – since we need to develop an effective assessment system for this medium in comparison with more traditional sources of wine information. I look forward to hearing your views.

But you might find their sincerity to be just a bit far-fetched, given the fact that the comment section under said posting inviting critique, is closed.

Ours is open – comment away!

Cheers!

The WineWonkette

***UPDATE: On Friday, February 4 when it first appeared on Wine Intelligence’s website, the Comment Section under the post Bloggers Bite Back inviting feedback was closed. Since the posting of this story and its appearance on a number of social media and on-line publications, the comment section is now OPEN.

About Amy Corron Power

A licensed attorney, Amy is a wine-lover, foodie, photographer, political junkie and award-winning author who writes about Wine, Food, Beer & Spirits. As Managing Editor & Tasting Director for Another Wine Blog, she travels all over the world's wine regions to share her experiences with her readers and nearly 10,000 twitter fans. Amy holds certifications through the International Sommelier Guild, and is also certified, with honors, as a California Wine Appellation Specialist (CWAS). She is a member of the Guild of Sommeliers, The Wine & Food Foundation of Texas and regularly attends Houston Sommelier Association events.
  • Pingback: Tweets that mention Wine Intelligence admits Bias, Ulterior Motives in "Wine Blogger Distrust" Release | another wine blog -- Topsy.com

  • http://twitter.com/palatepress Palate Press

    Oh Lord, the whole thing is crap. Compare YOUR wine store, the guy you go to, to “blogs,” and what do you think the result is going to be? Also, EVERY list of influential wine blogs based on readership is crap. CRAP CRAP CRAP. I have internal numbers for more than 100 different wine websites, numbers from two or three different counter embedded on every site, and the number in the publish sphere, and yes, I might be repeating myself, are CRAP CRAP CRAP.

    The bottom line? People trust people who repeatedly give them advice that pans out, whether it’s on the internet or in the store. Be honest with your customer, whether they’re a reader or a wine buyer, tell them what’s good, what’s bad, and what’s reasonably priced, and they will come back to you.

    As for the survey, it was so laughable from “GO” that I really didn’t give it a second thought. It was just another dying gasp of a threatened industry.

    • http://www.anotherwineblog.com WineWonkette

      But what do you really think? ;) We get different readership numbers from every analytical site we go to. But I just love a survey that provides a set list of wine “blogs” (many of which aren’t blogs at all) and doesn’t leave a space for “OTHER” in case their discrete list of 35 doesn’t cover it.

      • http://twitter.com/palatepress Palate Press

        No kidding.

    • http://www.anotherwineblog.com Houstonwino

      Crap! Definitely crap! ;)

  • http://twitter.com/Sedimentblog Sediment blog

    “You mean we don’t need to shut down our blogs and get a gig for peanuts…”?

    If you’re earning more than peanuts for wine blogging, then you’re doing well, sir!!

    • http://www.anotherwineblog.com WineWonkette

      Well I suppose even peanuts is better than no peanuts at all!

  • http://www.BATchat.net Steve Raye

    Bottom line, the methodology was flawed, the conclusions conflict with reality, and the whole thing is a bunch of hooey. I’m pulling together some of the research we’ve got access to and will be posting a more reasoned response on our blogs. Unfortunately damage has been done. I’ve gotten lots of calls and emails, especially from folks in Europe basically saying “I told you so…the Wine Intelligence report confirms that this social media thing is a fad. But it’s like saying research proves research works. Huh?

    The “nattering nabobs of negativism” (old Spiro Agnew quote I always wanted to use!) are too quick to embrace research that reinforces previously held misconceptions.

    There’s a slide I use in my social media presentations of a guy in a suit with his head buried in the sand on a beach. Up till now, I thought that was a pretty good visual metaphor for the “don’t get it yet” crowd. But this is not a matter of the tide coming in, the ocean levels are rising and the shoreline won’t ever be the same. Saying it ain’t so is a pretty feeble response to reality. Reminds me of Bush 2.

    • http://www.anotherwineblog.com WineWonkette

      Love your analogy Steve. And having worked in PR myself, I can tell you how easy it is to push a story with a great release from a trusted source — no matter what is in the release, as well as manipulate data to support your hoped for conclusions just in the way you ask the questions.

      Wine Intelligence as well as some of the folks on our side of the pond who continue to promote the idea that an independent writer is not to be trusted might take a look at where the whole wine-writing for the common man began:

      Robert M. Parker, Jr. started with a newsletter.

    • http://www.anotherwineblog.com Houstonwino

      While not much of a fan of Spiro T., I always loved that quote too.

  • http://twitter.com/WineHarlots Wine Harlots

    What a waste of time.
    The credibility and trustworthiness of Harpers and Wine Intelligence (whoever the hell they are) has diminished. Although it has inspired Wine Harlots to get into the survey business. We’re thinking of name our new biz Emperor’s New Clothes.

    • http://www.anotherwineblog.com WineWonkette

      Sign me up! But shouldn’t we call it the Empress’s? ;)

      • http://www.anotherwineblog.com Houstonwino

        The Harlots New Clothes has a certain je nais se quoi to it.

        • http://WineHarlots.com WineHarlots

          Deal Amy! It’s now Empress’ New Clothes.
          Joe, I love it, but with the Harlot, clothing is optional!

          • http://www.anotherwineblog.com Houstonwino

            As, so the story goes, was the Emperor’s. Also, the Harlot’s New Clothes won’t trigger Sinead O’Connor to start singing in my head. :)

  • http://www.vintuba.com ChrisO Vintuba.com

    Great article! Thanks for laying out all the evidence. Every report and survey ever written is subject until validated or refuted by a second or third source.

    Cheers!

    PS this would have made a great Mythbuster episode!

    • http://www.anotherwineblog.com WineWonkette

      Thanks Chris! And might I add that no poppy seed muffins were consumed while writing this post ;)

      • http://www.anotherwineblog.com Houstonwino

        And the toilet seat was down at all times.

  • http://www.winetonite.com Ed Thralls

    I had a boss who created his own version of “apples n oranges” that I think is more in line with this: “it’s like comparing dog sh*t and lime juice… the two just don’t go together.”

    Pathetic… nice work of analyzing the situation and sharing with us, AWB!

    • http://www.anotherwineblog.com WineWonkette

      Haha! Nice analogy. I can’t imagine anyone would enjoy that cocktail! Thanks so much for stopping by.

  • http://www.thevinofile.net Scott

    Nice analysis Amy. It seems to me that people trust recommendations from those they have built a relationship with. Traditional wine media gets the benefit of their ‘established’ relationship with the industry as a whole, making consumers trusting. Wine retail folks also get to build a relationship over time with their customers. Bloggers on the other hand are relatively new to most people, except for those who have caught on and follow specific bloggers religiously. I think you hit it on the head when you indicate that nearly 20% of respondents now say they trust the opinions of an independent blogger. The numbers to me are indicative of a big change, not of a sense of distrust.

    • http://www.anotherwineblog.com WineWonkette

      You’re right, since when does a 20% market share indicate “distrust”?

  • Pingback: wineconversation.com » Blog Archive » Wine research through a distorted lens

  • http://www.sommelierjournal.com Phil

    I agree with many of your problems with the survey, but your critique runs aground in a few areas:

    Sampling: The key pillar of scientific research is the ability to sample, so the fact that you’re only asking 1,000 people isn’t significant, that’s actually a pretty big sample. Think about political polls, that would be a huge sample for one of them, you must pay attention to the margin of error when thinking about the results, but statistically this is not an issue. Of course sampling has to be done correctly, and as you point out, geographically limiting yourself would be a big no-no.

    Elimination of people who deliberately didn’t take the survey (or as you put it, getting rid of Internet savvy folks): This is best practice behavior. Think about it this way, I have a survey that I know it takes at least 5 minutes to thoughtfully answer and takes at least 3 minutes to actually read. What do I do with someone who answered in 2 minutes 30 seconds? It is literally impossible to have actually read the entire survey in that amount of time, much less think about answers. Now how about someone who answered questions in a highly contradictory fashion or gave the exact same answer to all the questions, again, someone who has clearly not taken the survey? All of these people’s responses are tainted and need to be removed from the sample.

    Survey Quotas: It’s wrong to compare the quota’s created to the overall US population, you need to compare to the wine drinking population to see if there is a problem or is some demo is over or under represented. Structuring the quotas to gear toward the overall population would in fact be a mistake.

    So definitely room for improvement on the reporting side on Wine Intelligence’s end, and also issues with some of the questions, but you’ve gone a little too far in some of your critiques.

    • http://www.anotherwineblog.com Houstonwino

      Hi Phil, thanks for stopping by. I do have to disagree slightly with you here. Mainly because nitpicking this critique, which frankly is the only way to defend this admittedly biased and inaccurate junk study, implies that it is not an accurate criticism. You are correct that some collected data will need to be culled so as not to skew the results. However, if one wanted to skew the results, culling data is an excellent way to do so. But without seeing what was kept and what was tossed away we can only speculate as to whether it changed the results or not. Given their other tactics used to arrive at their conclusions, I think a little speculation can be forgiven on such a small point. As for sample size, you are correct. You are also correct that the geographic limits render that a moot point. All-in-all, Amy’s points are much more fair and accurate than this cynical study. To borrow the word coined by our illustrious collegue David, this CRAP is indefensible, so while you do have some vaild points, to a point, in the bigger scheme of things they are quite minor quibbles. Particularly given that the author of this post is a lawyer, not a statistician. She made her case quite well within her millieu, how do you think that the survey-takers did within theirs?

      • http://www.sommelierjournal.com Phil

        By the way, I think Amy is absolutely right, and I think bloggers in general should be much more focused on, the point that the data in this study, however flawed, shows that bloggers are becoming an increasingly trusted source, even among those who really don’t use them. What the actual numbers are I don’t know, probably not as low as this study would indicate (even better, from a blogger point of view).

  • http://www.sommelierjournal.com Phil

    Shoot, my reply to Houstonwino seems to have gotten overwritten by my PS, so here is the condensed version:

    Reason I think what I wrote is important is because I don’t think it does anyone any good to have misconceptions about the basics of marketing research percolating around the Internet. In my view, Amy was wrong in what she wrote that I responded to, which in my view were significant parts of her post, since they had their own numbered sections.

    Removal of respondents could be sinister if you are attributing sinister motivations to Wine Intelligence, but at that point pretty much anything is possible as you’re basically accusing them of fraud.

    Amy’s profession doesn’t matter because the post is supposed to be proof that Wine Intelligence had bias and ulterior motives in their study in part proved by thorough examinations of their methodology, which forfeits the right to say “this isn’t my area of expertise.”

    Ultimately I believe we’ve got GIGO at work here (garbage in, garbage out) from Wine Intelligence, not a plot to discredit wine bloggers.

    • http://www.anotherwineblog.com Houstonwino

      I’ll let Amy speak for herself and her facts. She used to be in marketing and will assuredly want to address your points. However, my mention of profession was merely to point out that she is inclined to build a case, which she did pretty convincingly. As your last statement shows, we are in agreement that Wine Intelligence failed to make theirs. Point being, if she made a mistake or misspoke (miswrote?) or was even unclear in a point, it becomes the equivalent of an Apple fanboy arguing the semantics of the word “open” as applied to Android when the subject is really that Apple is completely closed. (I figure with GIGO reference that might be an appropriate analogy;) )

      I don’t know what motives WI had for publishing this turd, but whether it was out of malice or just laziness, it has the potential to do considerable damage to a lot of people. Their admission of intentional bias to sell copies might even make this thing criminal. <– The legal opinion of a web developer who didn't even sleep at a Holiday Inn last night.

      Anyway, I think we basically agree on everything but a point or two, so I am going to let this drop and go open a bottle of something. Cheers! :)

  • craig

    I wonder if the Wine Intelligence report said 66% of people trust bloggers if you would have disected the report witrh as much vigour, & indeed vitriol.
    While the methodology may be less than perfect, most surveys would suffer the same less than perfect methods.
    Attacking the report does nothing for the blogesphere credibility, instead build on the 22% so the next report may say 32%, and so on, as time goes by.

    • http://www.anotherwineblog.com Houstonwino

      Thanks for *ahem* contributing to the conversation, Unregistered Craig (with no e-mail address). Your comment, well, added to the total number of comments. That’s something, I guess.

    • http://www.anotherwineblog.com WineWonkette

      Someone at Wine Intelligence had the cute idea to put forth an inflammatory, tabloid-type press release suggesting that independent wine writers were not to be trusted. In doing so he invited scrutiny. Had someone not dug into and exposed the methodology, lazy journalists everywhere would have gone on citing “vinifudge” as if it were “vinifact.” They still might.

      What they’ve done is put on question mark on their own credibility. If one puts forth an inflammatory press release alleging “distrust” of a group of independent writers, one bloody well better have clean hands when it comes to the information he is peddling.

  • http://www.anotherwineblog.com WineWonkette

    My comments were based on experience in the PR and marketing research business. It boils down to oversampling, under-sampling and spin. I understand statistical sampling, and concur that one need not survey the entire population of regular wine drinkers to infer results from the data. But a sample needs to be representative to be meaningful. Wine Intelligence’s sample was not. Either negligently or recklessly, Wine Intelligence concentrated on only those living in a portion of the original 13 colonies, adding the city of Chicago, and leaving out not only the state that actually produces the bulk of wine in the U.S., but that also is the home to the most forward thinking technology companies on the continent.

    Perhaps they don’t understand the geographic and demographic make-up of the U.S (they certainly don’t seem to understand that Washington, D.C. is not a state). And perhaps they didn’t put enough thought into “what is an independent wine blog.” If this is the case they have no business charging clients $2,150 USD for a report with half-baked conclusions.

  • Pingback: Tweets that mention Wine Intelligence admits Bias, Ulterior Motives in "Wine Blogger Distrust" Release | another wine blog -- Topsy.com

  • Steve

    San Diego State University has an outstanding Business of Wine professional certificate program. The next class starts Feb. 15 and is taught by advanced sommelier Lisa Redwine (yes, that’s her real name:)
    http://lajolla.patch.com/articles/with-a-name-like-redwine-she-has-to-be-good

  • http://www.1winedude.com 1WineDude

    Be careful with the cellarer.com data, folks. It’s also, well, to quote David, CRAP. :)

    • http://www.anotherwineblog.com Houstonwino

      Hey Joe, what is wrong with Cellarer’s data? I just gave it a very cursory look and nothing jumped out as obviously wrong, but I don’t have the time right now to do an in-depth analysis of the methodology right now. But if we ever have to do another one of these type of stories I don’t want to use crap to refute other crap. There is enough of that horseshit in the world already. E-mail me if you’d rather not get into that on here.

      • http://www.1winedude.com 1WineDude

        David can probably back me up on this, but the data sources used by Cellerar are largely inaccurate. Compete.com for example does not provide accurate traffic numbers (they are estimates) and that’s a major source of the Cellar calculation.

  • CassidyGrapeplanting

    Great Post! Very cute too actually!

  • http://pulse.yahoo.com/_ISRT2BNNJ2GE7JAJXSPNREC6DM Fanny Wang

    As a part of the lighting system, head lamp is very important to the car driving procedure. Electronic test instrument is used to create signals and capture responses from electronic devices. The ignition distributor spreads electricity to the proper cylinder. The timing gear is connected by chain, gears or a belt to the crankshaft at one end and the camshaft on the other. Brake drum is a type of the brake system. Nowadays there are various types of car handsfree products for customers’ choice.

  • Pingback: Announcing the Shortlist for 2012 Born Digital Wine Awards | Born Digital Wine Awards

  • Pingback: Winedr Blog » Born Digital Wine Awards 2012 Shortlist

  • Pingback: Amy rides the short bus! | Another Wine Blog

  • Pingback: 2012 Born Digital Wine Awards Winners | Born Digital Wine Awards