MediaLife: More snake oil from DoubleClick

Paul Benjou, director of client services at AdWare (a ValueClick company), writes with skepticism about a new piece of research about cross-media marketing effectiveness from DoubleClick, which we reported on yesterday. The opinion piece basically questions the methodology of the research.

A bunch of industry folks on a private discussion forum I belong to are bent out of shape by the opinion piece, pointing out, among other things, that Mr. Benjou bases his criticism simply on the press release, not on the actual research. I'm with them. Click "MORE" for my hearty rant on the subject.

I have no insight into this particular study, but as a professional researcher, it always annoys me to see armchair pundits taking seemingly gratuitous pot-shots at industry research. For example, he raises lots of questions about the methodology, thereby impuning its credibility, saying "it encourages the application of twisted logic," yet he apparently never lifted the phone to put those questions to the study's authors to get actual answers to his questions. I'm a big fan of questions, but only when they lead to answers, not just sarcasm.

I know too well from first-hand experience that researchers can invest many, many hours in creating sound (or at least well-intentioned) research only to have the PR folks overly simplify the conclusions in a press release that becomes the basis for everyone's reaction to the study.

I don't mean to suggest that we shouldn't be self-critical, but often I think such commentary is more about picking a convenient topic for one's column deadline (and being snarky about research is always easy fodder), which loses sight of the big picture.

Here's the big picture: this industry needs to demonstrate to traditional advertisers that they should shift more budget online. We need to convince them of that because A) we all work in the industry, and a rising tide lifts all ships, and B) because we all believe that to be true (based on a growing body of research and our own shared experiences). We're in the trenches. We know, or heartily believe, anyway, that online marketing works. Yet, when we spend our energy bickering over whether online is better for DM or branding or picking apart the methodology of a particular piece of research (here's one example*, here's another*, here's another, and here's another), all we succeed in doing is looking divided and unsure about the effectiveness of the medium to the one audience that really matters: skeptical offline advertisers. (In the interest of full disclosure, two of these pieces of research that were up for criticism were ones I authored, as noted with the asterisk *.)

Of course I'm not saying that all research should be swallowed hook, line and sinker, that we shouldn't dare question authority or attempt to keep ourselves honest. It's just, do these critics really believe that people are not paying for online content, or that Google isn't a huge brand, or that cross-media marketing does not work, or that email is not effective for branding, or that there is not value for advertisers in affiliating their messages with premium content (the subjects of the various research being criticized in the links above), etc.?

No. They're all e-marketing professionals themselves. In all of these cases I've linked to, the critics (not one research professional among them) are not saying that the conclusions of the given studies are false (i.e., that various aspects of e-marketing are effective). Instead, they are simply nitpicking over the definition of terms or subtleties of the methodological approaches or (incredibly, in two of the cases) that the conclusions of the research are so obvious (that email is effective for branding and that users with an affinity for a particular site will have a higher affinity for the ads on that site) that the questions should never have been asked in the first place.

I am a curious person, which is why I've made my career as a researcher and journalist. I do believe there is no such thing as a question not worth asking, particularly in an industry as young as ours. But I'll let you in on a little secret about the research industry: there is no such thing as a flawless methodology. Real life is not a petri dish. You want to talk about flakey research? Just take a look at the methodologies that have built traditonal media: TV viewership panels, reach and frequency estimates, day-after ad recall, reader surveys, circulation audits, etc. You could drive a truck through the holes in those methodologies.

The point is, such research is directional, not absolute. The key question is really "is it bigger than a bread box?" Maybe the "true" answer is really 6%, maybe it's 8%, maybe it's 10%, but what's valuable to know is that it's not 80%.

In college, I took a course called "The Philosophy of Meaning," where the professor made an interesting point: truth is not any more valuable on a moral scale than falsehood. Her example: a fireman asks a bystander how many people are in the burning building. In fact, there are 24 people in the building. The inaccurate answer 25, however, is a lot more valuable to the fireman than the accurate answer "less than a million." The value of information is in its interpretation.

Is cross-media marketing effective? Yes. I base that judgment not on the conclusions of this one particular study, but on the balance of lots of research and anedotal evidence that suggest that is the case. Is the methodology of this particular study airtight? I don't know, but if I actually wanted answers to the questions about the methodology that Mr. Benjou raises, I would have put them directly to the study's authors (or, for starters, sought out a copy of the actual study, not just the press release, to see whether those questions are addressed in a methodology section) rather than simply labeling the study "snake oil" in a public forum.

No single study in this or any industry should be seen as "proof" of anything. But in aggregrate, each piece of research incrementally builds a body of evidence that points in the direction of the overriding truth: the Internet is bigger than a bread box.

So, next time you feel like railing about the soundness or value of an industry research study, why not save it and instead write something positive, informing the ignorant and non-believers as to what online marketing strategies actually do work. This industry needs more information, more best practices and more examples of how to do it right, not more cynicism. Particularly not from the team.