Lies, Damned Lies, Statistics, and Metrics

Mark Twain popularized (and attributed erroneously, ironically, to British Prime Minister Benjamin Disraeli) the adage that there are three kinds of lies: lies, damned lies, and statistics.

This week, my brilliant friend Jason shared with me a piece in The Correspondent entitled The new dot com bubble is here: it’s called online advertising. He in turn got it from Dr. Michael Burry’s twitter feed (of Big Short fame). It’s important to me that everyone receive their due credit for bringing this article to my attention, because it had a profound impact on my thinking, and I have been sharing it with just about everyone and anyone who will listen all week.

I would like to humbly suggest that there is a fourth category of lies to be added to the list: metrics.

A statistic can be dishonest (in this rhetorical and liberal application of the concept of a lie) because it can be used to misdirect. A statistic, however accurate, can mislead its audience by appearing to be representative of something that it is not; Dr. Stephen Jay Gould does a good job demonstrating this phenomenon in his 1985 essay The Median is not The Message, as does SMBC in the comic below:

A metric, on the other hand, as described by Frederik and Martijn in the Correspondent, isn’t misused to misdirect from the truth, but rather to drown the truth in a sea of meaningless truths, and ascribe false importance to unimportant things.

There’s an old joke about 3 secret agents, one from MI6, one from the CIA, and one from the KGB, who upon meeting at a resort in the alps place a wager to determine who is the best at apprehending persons of interest.

They release three rabbits into the woods, to see who can retrieve their rabbit the fastest.

First, the MI6 agent goes in to the woods, and after five hours he returns to the resort with his rabbit.

Then the CIA agent goes into the woods, and after three hours, he returns to the resort with his rabbit.

Finally, it’s the KGB agent’s turn. He returns to the resort 20 minutes later with one of the waiters from the resort, bruised and bloody, missing several teeth, with his hands on his head, crying out, “okay, I admit it, I’m a rabbit!”

The most common reaction that I’ve received from folks (especially marketing professionals) with whom I’ve shared the Frederik and Martijn piece in the Correspondent – which you really, really ought to read in full – is that the arguments made in this article resonate immediately and powerfully with them. These are people, most of whom have been on either side of the table when digital ads are pitched as a cure all; they immediately recognize a thorough and quantifiable case for what they’ve long suspected. They’ve felt uneasy about the promises they’ve been making, but these guys have the receipts.

The piece makes a compelling case that people who sell digital marketing based on metrics – hard numbers that prove their effectiveness – are largely confusing correlation and causation.

The people who are buying the ads are being offered solid proof that they work in the form of metrics. What they’re being given instead is a bloodied waiter insisting that he’s a rabbit.

One friend of mine who works in digital marketing put it beautifully:

“Oh well the impressions are great which MUST mean purchase consideration has gone up… they will go and purchase via another channel… blah blah blah”

It is his firm’s job to convince the client, post hoc, that one of if not the primary reason that their sales went up, is because they paid an influencer an obscene amount of money in exchange for impressions. They can’t prove a causal link between the impressions and the sales, but they can certainly present a correlation as a causal link.

They are there to present the metrics – any metrics – in such a way that they justify increases in sales and excuse decreases in sales, as if any of the methods and tools that they are employing are employed in a vacuum.

There is a correlation between the strong results that digital marketing firms can show, and their proliferation as the preferred marketing method. In fact, nearly 40 cents of every dollar that venture capitalists invest in startups goes to Facebook or Google for ads.

See what I did there? That’s a correlation, but not necessarily causation. If I was a salesman at a digital marketing firm, you’d better believe that I wouldn’t be differentiating between the two.

Why am I writing this? Frederik and Martijn already did a much better job than I could taking the digital ad industrial complex out to the woodshed, and there are few things more cliche than writing a hit piece on a product that can be used as an alternative to your own.

I like digital ads just fine. We use digital ads. We have digital marketing contractors and firms that work on behalf of our brand, and we demand colourful graphs and pie charts just like everyone else.

I’m writing this because of what I refuse to give our clients and prospective clients, that nearly every prospective client has asked me for: the certainty of a number.

My prospects all want what their digital marketing firm is offering them. They all want metrics that show that their campaign is going to be a risk-free slam dunk. That you can run a business on autopilot with charts and graphs and bells and whistles. That there is a slot machine that you can put a dollar into and get 2 dollars out of.

And it’s not that I don’t have the numbers – it’s that I have too many numbers. I have the nonprofit who saw an immediate 16% increase in new donations from their first handwritten mail campaign, and the nonprofit who saw only a small discernible change in their acquisition numbers, stopped sending handwritten mail, and then noticed that the people who received handwritten mail donated in abnormal numbers and higher amounts over the following 9 months.

I have numbers from half a dozen political campaigns that have seen an increase in voter turnout amongst those that they’ve mailed handwritten letters to, and each of them contradicts the others. 2% increase, 5% increase, 7% increase, 15%  increase, and yes – no discernible difference. Which number should I take credit for? Which number should I own and sell to the next political campaign that asks for metrics?

What do I say to the financial institution marketing manager who played me a voicemail of someone thanking them for sending them a beautiful handwritten letter, but telling them that regrettably they were still on the fence about the product. Someone had felt moved by the product to reach out proactively and tell them that they were still undecided. What metric do you file that under? Customer engagement? Positive brand association? Our product clearly nudged their clients closer to making a massive purchase, but I can’t sell a fraction of an acquisition to the next prospect who asks.

Nor would I. Because impressions are not clicks, clicks are not conversions, and conversions don’t live in a vacuum.

I have political candidates who can provide me no end of anecdotes of people saying that they voted for them because they received a handwritten letter. I have nonprofit CEOs who measure the success of a handwritten campaign by the feedback they get from their mother’s friends.

But most of all, I have prospects asking to be lied to in just the right way so that it seems like I am backing up my claims with indisputable facts.

The truth is that handwritten mail works. It works for acquisition, it works for retention, it works for engagement, it works for mobilization. It works better than printed mail, and email, and bench ads, and Instagram influencers.

Sometimes it works slowly and sometimes quickly. Sometimes it works in concert with online marketing efforts, and sometimes it stands alone. Sometimes it inches your client towards the sale, but doesn’t get them all the way. Sometimes they make their purchase a week later through a different channel. Sometimes it lands in the mailbox of a client who is already close to buying, and it appears that the handwritten letter worked like magic. Sometimes it makes it harder for donors to leave – donors who you never knew were thinking about leaving. Sometimes it gives people warm, fuzzy feelings about your brand that translate into a big purchase, a large donation, an important referral, or a vote. Sometimes it makes people decide, in the back of their mind, that they’re going to say yes to your next ask – an ask that comes a year later; or that they’re going to refer the next friend who is in the market to purchase their house with you.

I spend a lot of my time with clients learning about how it works, understanding and writing about why it works, and most of all – refusing to quantify how much it works.

There are things that have a huge impact that can’t be measured. There are also things that can be measured in such a way that they appear to have a huge impact, but don’t.

The movie version of The Big Short begins by ironically mis-attributing another extremely relevant quote to Mark Twain – one that should come as a warning label with every digital marketing pitch:

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.”