Copy Testing — Predictive Magic Or Paper Shield?

Illustration by Will Maynard

Fans of The Office will know this one: Michael Scott is driving while Dwight Schrute sits shotgun. The GPS tells them to make a right, obviously instructing them to bear right a little further up the road. Refusing to take the directions in context, Michael immediately makes a hard right — straight into a lake. “The machine knows!” He screams, as the car slowly begins to fill with water. (And, no, he didn’t get rental insurance, because this is Michael Scott.)

In advertising, that’s not a bad metaphor for copy testing, says Martin Agency CSO Elizabeth Paul. Depending on whom you ask, pre-testing is either pragmatically essential or painfully endured.

Some marketing organizations are prohibited from moving work into production (or into media) without a “score” validating its wisdom and (theoretically) predicting its success.

It’s easy to understand why, Paul points out: Many companies see marketing as a cost center rather than a growth driver. CMOs often find themselves having to justify every penny, and the best way of keeping their budgets is to be able to promise the return on investment. Programmatic has the allure of offering minimal investment for measurable output. Who wouldn’t want the same for every part of the funnel?

And for every uncertain marketing organization, there is a black box methodology that promises predictive scores weighed against a body of category averages, says Paul. She equates this to paint-by-numbers, aka best practices promising that if you “Brand early!” and “Convey information!,” you’ll execute a winner — guaranteed.

Of course, this proposition can be intoxicating to brands. The challenge is that there is no system for mitigating risk in creative — because all creativity includes risk, says Paul. Industry lore is littered with examples of iconic work picked apart in testing, as well as brave marketers operating from a place of prophetic vision in the face of difficult findings.

“It’s impossible to talk about testing without talking about fear,” says Paul. “Testing can be a paper shield: a shield from criticism, a shield from getting fired. Some organizations are so afraid to be wrong, they’re unwilling to risk being right.”

Copy Testing Conundrum

Relying on copy testing can lead brands to knee-jerk reactions, scrapping anything that doesn’t mesh with the test results. While this can seemingly lead to safe ideas, it can also lead to middle-of-the-road ones, too. (Hence why 84 percent of ads are effectively invisible, as per an Ehrenberg-Bass Institute for Marketing Science study.)

Part of the problem with relying on copy testing is that audiences often can’t tell what ads they’ll connect with until they actually see them organically in action. “Human beings are not great at accurately predicting what they’ll do in a given situation, or why they like what they like,” says Paul. “You're asking people to rationalize what they find motivating, and the body of research on human behavior says we're not very good at that.”

This is one of the reasons why GEICO believes creative testing should play a minor role in making video content decisions, says Martin SVP/Group Account Director Matt Mattox. “It switches people from consumer to evaluator,” he says of testing. “It forces an artificial and rational response to what, in the wild, will be a very different interaction — one that needs to be entertainment forward to have a chance at succeeding.” (While the brand does test some spots post-production, it does so to stay updated on consumer interests and long-term trends across campaigns and executions, says Mattox, calling it more of a “guardrail” than a “decision point.“)

He points to seminal GEICO spots like “Hump Day” and “Scoop! There It Is!” as examples of work that might not have tested well — after all, camels and ice cream seemingly don’t have much to do with insurance. But the spots were huge hits when presented to audiences in the real world, because they convey information in an engaging, unexpected way.

One reason GEICO is so successful when it comes to marketing is because it doesn’t follow the prescribed, formulaic sales messaging that testing would have marketers believe is essential, says Mattox. Instead, GEICO believes that if it entertains people by delivering great creative work, they will like the brand, be more willing to listen to and believe the sales message and, critically, purchase.

“GEICO’s scripts are incredibly hard-working,” points out Mattox — they deliver both key information and entertainment. The trick is finding the right balance between the two that lets the creative make an impact. “Without that, the magic is quickly lost and the work becomes invisible, even if it’s saying all the right things and has a ton of media weight behind it,” says Mattox. “Over decades of working together and hundreds of spots, we’ve realized it’s best for both GEICO and us to determine that balance, instead of asking testing to tell us.”

Another issue with relying on copy testing is that clients are often releasing ads based on numbers that are months old. This can be a blow to a brand in a world where the zeitgeist rapidly changes every day and cultural relevance is essential.

”I don't think that the methodology has kept up with the various channels that we have. I have not seen how you test a TikTok ad, for instance,” says Nancy Hill, former 4A’s CEO and founder of the group Media Sherpas. “I think the testing that's being done is still being done primarily for traditional media. And, you know, that's been based on norms that are 20, 30, 40 years old.”

That’s another benefit of GEICO not relying on testing or strict parameters for creative work, says Martin VP/Account Director Jon Glomb — it allows the brand to fluctuate and grow organically alongside culture. While GEICO does have brand guidelines, they operate more like buoys, rising and falling with the tide to remain pliable. Glomb points to the evolving nature of GEICO’s humor as an example — if you look at a spot from 15 years ago, it will be a slightly different type of humor than used today. That’s not because the brand has gone off-course; it’s because it’s kept pace with culture as humor has shifted.

”This approach creates both continuity and longevity for the brand,” says Glomb. “If the brand guidelines are too rigid, culture will move beyond the brand, and then it won't be relevant anymore. Many brands fall victim to their own rigidity, and once their relevance wanes, it can necessitate a sweeping identity refresh that could and should have been avoided.”

Analysis Without Paralysis

Copy testing won’t provide all the magic answers. But that doesn’t mean that ad professionals should eschew all data and audience insight when creating work — they just need to rely on the right kinds of data and insight at the right time, says Paul. “When we get into copy testing, the conversation gets framed as ‘creativity versus science,’” she says. “That’s the wrong conversation; it has to be both.”

For instance, advertisers can target niche demographics within their customer base and gather key information on the cultural moments and topics that resonate with them, such as what they share and talk about on social, or the events and influencers important to them.

Similarly, real-time testing lets brands see how an audience reacts to a campaign as it unfolds, which makes for quick market optimizations as feedback arrives. This also grants brands more flexibility, as opposed to putting all their eggs in one basket based on testing results. And fields of study, like behavioral economics, provide psychological and scientific lenses into how consumers think and why they make the choices they do. (For instance, Nobel Prize winner Daniel Kahneman’s theory of System 1 vs. System 2 thinking.)

And advertisers can also look to feedback from within their own team. Foregoing copy testing doesn’t mean just relying on intuition or hunches, says Paul; rather, advertisers can rely on expertise — a combination of collective experience and research.

Leaning on expertise rather than copy testing leaves enough space for imagination to thrive without abandoning data. “There's something about creativity that is never entirely certain, because you're forging something new,” says Paul. “But it can be informed.”

And expertise allows advertisers the confidence to make hard decisions and take chances on the creative unknown: “You can’t take bravery out of success,” says Paul.

Case in point: While GEICO doesn’t use testing to make its creative decisions, that doesn’t mean it’s operating on a whim. Instead, its creatives, strategists and account people pool together their combined business experience and deep knowledge of the brand — aka their expertise — to figure out what ideas might work. This leads to successful creative work because the team knows the ethos and voice of the brand without trying to overly control and standardize it.

“A lot of times, well-intentioned marketers want to domesticate their brand like a lab — sit, roll over, lay down,” says Mattox. “Whereas at GEICO, we treat it more like a wolf. It’s a wild animal. We commune with it and we understand it, but we don't tell it what to be like. We let it lead us.”