How to Read Critical Feedback on Your Ads

This is a guide, not a case study. We are not going to show you a brand that turned criticism into a 340% lift, because we do not have one and neither does anybody who has not shown you their method. RateAds is in public beta; we make no claims about outcomes anyone has achieved here.

What follows is a way of reading honest, sometimes harsh, public feedback on your advertising: what it is actually evidence of, what it is not evidence of, and how to change your work in response without over-correcting. Where a claim rests on published research, the paper is cited and linked at the end.

Start here: the feedback you get is not a sample of your audience

The most expensive mistake advertisers make with public feedback is treating it as a survey. It is not one, and the reasons are well documented.

Schoenmueller, Netzer and Stahl compiled over 280 million reviews across 25 major online platforms and found that most reviews on most platforms show strong polarity: a mass of ratings at the positive end, very few in the middle, and a cluster at the negative end (Journal of Marketing Research, 2020). Their identified driver is polarity self-selection — people with extreme evaluations are simply more likely to write something. Crucially, they show this polarity reduces the informativeness of the review distribution. The shape is not telling you about quality; it is telling you about who bothered to speak.

Hu, Zhang and Pavlou had reached a compatible conclusion earlier (Communications of the ACM, 2009), attributing the J-shaped distribution to two self-selection biases: a purchasing bias (the people who engage are already unrepresentative) and an under-reporting bias (moderate experiences go unreported).

Brandes, Godes and Mayzlin added a third mechanism, differential attrition — potential reviewers with moderate experiences are more likely to drop out of the pool of active reviewers than those with extreme ones (Journal of Marketing Research, 2022). In a large field experiment with an online travel platform, an unincentivised solicitation email measurably reduced review extremity, which is direct evidence that who you ask changes what you hear.

Three independent literatures, one conclusion for you: the loudest signals in your feedback are the least representative ones. Your job is not to average the comments. It is to work out which comments are describing a real property of the ad.

Sort comments into four piles before you decide anything

Not all criticism is the same kind of information. Before you argue about any of it, sort it:

1. Comprehension failures. "I watched the whole thing and I do not know what you sell." "I thought this was for insurance." These are the most valuable comments you will ever get and the least ego-flattering. Comprehension is a property of the ad, not a matter of taste, and if several unrelated people independently missed the same thing, they are right and you are inside the curse of knowledge. Act on these almost unconditionally.

2. Craft observations. "The logo appears once, in the middle, and I looked away." "The text is too small to read on a phone." These describe execution and are usually checkable against the artifact itself. They are frequently correct even when rudely phrased. Act on these when you can verify them.

3. Taste. "I hate this song." "The humour is not for me." This is real information about that person and weak information about your audience. It aggregates poorly and it is where over-fitting starts. Log these, do not chase them.

4. Values objections. "This portrayal is demeaning." "This claim is not true." These are not taste, and treating them as taste is how brands get into trouble. Substantiate or fix — separately from the creative discussion. Escalate these; do not average them into a score.

Most arguments about feedback are really arguments about which pile a comment belongs in. Do that sort first and the disagreements shrink.

The craft complaints that research supports

Some of the most common criticisms map directly onto findings in the attention literature, which makes them easier to take seriously.

"I did not notice your brand." Pieters and Wedel eye-tracked 1,363 print ads with more than 3,600 consumers (Journal of Marketing, 2004). The pictorial captures attention largely independent of its size; the text element captures attention in proportion to its surface size; and the brand element is the most effective at transferring attention onto the other elements. If people are absorbing your image and missing your brand, making the logo bigger is not obviously the fix that the research points at — the brand's role is to hand attention onward, and only increases in the text element's surface produced a net gain in attention to the ad overall.

"I skipped it." Teixeira, Wedel and Pieters modelled moment-to-moment avoidance using zapping data and eye tracking on 31 commercials with nearly 2,000 participants (Marketing Science, 2010). Simple measures of attention dispersion strongly predicted avoidance. Separately, central on-screen brand positions promoted avoidance — brand size did not. Their proposed remedy was brand pulsing: spread the same total brand exposure through the spot instead of concentrating it.

So when a reviewer says "it felt cluttered and I bailed," they may be reporting a measurable property of your edit rather than a mood.

"It was funny but I still do not know what you do." Eisend's meta-analysis of 369 correlations found humour reliably improves attitude toward the ad, attention and positive affect — and also that it significantly reduces source credibility, with the effect on ad attitude roughly twice the size of the effect on brand attitude (Journal of the Academy of Marketing Science, 2009). A reviewer who liked the joke and distrusted the claim is describing exactly the trade-off the literature predicts.

How to avoid over-fitting to a vocal minority

Six habits that keep feedback useful instead of destabilising:

Count independent observations, not comments. Five people quoting each other in a thread are close to one observation. Two strangers on different days raising the same confusion are two.

Require a mechanism. Before changing anything, be able to say why the change would fix the complaint. "The product does not appear until 0:22, and three people said they did not know what was being sold" is a mechanism. "People did not like it" is not.

Do not rebuild a campaign on a handful of comments. Even large, properly randomised experiments struggle to resolve advertising effects: Lewis and Rao found the median confidence interval on advertising ROI across 25 large field experiments was over 100 percentage points wide (Quarterly Journal of Economics, 2015). If millions of impressions leave that much uncertainty, nine reviews will not settle a strategy question.

Distrust your own attribution report. Gordon, Zettelmeyer, Bhargava and Chapsky compared observational estimates with randomised experiments across 15 Facebook campaigns — 500 million user-experiment observations, 1.6 billion impressions — and found observational methods often fail to recover the experimental effects (Marketing Science, 2019). If your dashboard disagrees with a consistent qualitative signal, the dashboard is not automatically the adult in the room.

Do not chase the extremes back toward the middle. The polarity research says your one-star and five-star writers are the self-selected ends of a distribution. Designing to silence the one-stars usually means designing away whatever the five-stars responded to.

Set a review cadence, not a reaction reflex. Read feedback in batches on a schedule. A comment read the hour it arrives gets weighted by adrenaline; the same comment read alongside thirty others gets weighted by frequency.

On responding publicly

The evidence here is more specific than the advice usually is. Proserpio and Zervas studied hotels' use of management responses to reviews (Marketing Science, 2017). Responding hotels saw an average 0.12-star increase in ratings and a 12% increase in review volume. But there was a trade-off worth knowing about: once hotels began responding, they received fewer but longer negative reviews — the authors argue unsatisfied consumers become less likely to leave short, indefensible criticism when they expect it to be scrutinised.

Two caveats before you generalise. This is the hotel industry, not advertising, and it is quasi-experimental rather than a randomised trial. What it supports is narrow and still useful: responding changes who writes and what they write, not just how you look.

If you do respond, the version that survives contact with a sceptical audience is short, specific, and non-defensive: acknowledge the concrete point, say what you are doing about it or why you are not, and stop. Do not argue taste. Do not imply the reviewer misunderstood unless you can show what they missed. Do not respond at all while angry.

What not to do

• Do not try to get honest criticism removed. A negative opinion is not a policy violation. Report content that is abusive, fraudulent, or off-topic — not content that is unflattering.
• Do not organise favourable reviews. Beyond being against the rules of any credible platform, incentivised or solicited-from-fans feedback re-introduces exactly the selection bias that makes review data hard to read in the first place. You would be paying to corrupt your own instrument.
• Do not treat a single reviewer as a research panel. One articulate person is one person, however well they write.
• Do not stop at the score. The distribution and the reasons carry the information; the mean of a polarised distribution carries almost none.

The short version

Public feedback is a hypothesis generator, not a measurement instrument. It is excellent at telling you what is confusing, what is invisible, and what is objectionable — the things people notice and can articulate. It is poor at telling you how much anything is worth, because the people who write are not the people who watched.

Use it to find the problems. Use experiments, when you can afford them, to decide what the fixes are worth.

Sources

1. Schoenmueller, V., Netzer, O., & Stahl, F. (2020). The Polarity of Online Reviews: Prevalence, Drivers and Implications. Journal of Marketing Research, 57(5), 853–877. doi.org/10.1177/0022243720941832
2. Hu, N., Zhang, J., & Pavlou, P. A. (2009). Overcoming the J-shaped distribution of product reviews. Communications of the ACM, 52(10), 144–147. doi.org/10.1145/1562764.1562800
3. Brandes, L., Godes, D., & Mayzlin, D. (2022). Extremity Bias in Online Reviews: The Role of Attrition. Journal of Marketing Research, 59(4), 675–695. doi.org/10.1177/00222437211073579
4. Pieters, R., & Wedel, M. (2004). Attention Capture and Transfer in Advertising: Brand, Pictorial, and Text-Size Effects. Journal of Marketing, 68(2), 36–50. doi.org/10.1509/jmkg.68.2.36.27794
5. Teixeira, T. S., Wedel, M., & Pieters, R. (2010). Moment-to-Moment Optimal Branding in TV Commercials: Preventing Avoidance by Pulsing. Marketing Science, 29(5), 783–804. doi.org/10.1287/mksc.1100.0567
6. Eisend, M. (2009). A meta-analysis of humor in advertising. Journal of the Academy of Marketing Science, 37(2), 191–203. doi.org/10.1007/s11747-008-0096-y
7. Lewis, R. A., & Rao, J. M. (2015). The Unfavorable Economics of Measuring the Returns to Advertising. The Quarterly Journal of Economics, 130(4), 1941–1973. doi.org/10.1093/qje/qjv023
8. Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook. Marketing Science, 38(2), 193–225. doi.org/10.1287/mksc.2018.1135
9. Proserpio, D., & Zervas, G. (2017). Online Reputation Management: Estimating the Impact of Management Responses on Consumer Reviews. Marketing Science, 36(5), 645–665. doi.org/10.1287/mksc.2017.1043