Re:filtered #17: The promise and peril of AI-powered audience insights

Over the past months, I've studied AI tools for audience research – for presentations, pitches, our own research projects, and many conversations, some heated.

I've seen presentations filled with tons of amazingly promising generated insights, and predictive models. I've watched colleagues' eyes light up at the possibilities, sometimes mine did too.

But I also always felt a bit of dread that something was off, too good to be true, that the output often felt too smooth and simple to account for the complexity and messiness of the human experience.

It's the same dread I get when I hear someone confuse journalism activity for impact, metrics for meaning. There’s some flaw that hasn’t been fully conceptualized, at least in my mind. Maybe in building these systems, we're asking the wrong questions?

Below are rudimentary thoughts as I try to make sense of this. I'm still figuring it out, and I'd genuinely value your perspective on where I might be wrong or what I'm missing. (Reply to this email or message me on Signal.)

The repository graveyard

At Radio Free Europe, we had decades of audience research, gathering digital dust. So much work was involved, millions were spent, but largely unused. Why?

Leadership needed bigger numbers, and a record of regular measurement. Journalists pursue the latest “news” no matter what the research says.

Could AI finally help extract value from extensive research archives and data that many newsrooms undoubtedly have? Could it help us make it accessible to a newsroom? That was my first question, because any incremental gain in relevance was better than nothing.

The second came from the nature of research. In highly oppressive places, the standard sampling correctives of research don’t work: people self-censor, internalize oppression, give answers they think are expected in ways that traditional research can’t correct through, say, greater sampling.

I have not yet seen a lot of research on correcting for social desirability bias in audience research. If you know of any or have your ways, please let me know!

List experiments (in which respondents indicate the number of statements they agree with, not which ones) appear to be promising: This 2022 study in Tunisia showed that direct questions overreported support by 40-50 percentage points compared to list experiments.

In this 2018 study on Colombian counterinsurgency support, the preference falsification in guerrilla-controlled areas was slightly more than 50 percentage points.

By contrasting direct and list questions in China, this 2024 study found that citizens overstate their support for the Chinese Communist Party by an average of 28.5 percentage points, nearly three times higher than similar studies in Russia and double the average found in other autocracies.

There may be a deeper, conceptual challenge: there is no free/unfree world, freedom vs autocracy binary, we all live in multitudes of systems, many of which are oppressive to some degree.

Can we figure out alternative sources of information to build greater empathy for lived experiences anywhere (so that any civic-minded media can be relevant and useful)? Isn’t there a ton of information already out there?

When you can't reach people directly or can’t get information from them you need to articulate value propositions that they may be likely to respond to, you often have to look sideways - indirect sources, news articles, proxy populations, like former community members.

There's always some information somewhere, but surfacing it systematically and attributing the right levels of confidence to them can be really time consuming and hard. Could AI help identify alternative information sources? That was my second question.

In both cases, I've found AI to help so much. Not only does it make research quicker and tangible for the big shops, it actually makes it affordable and possible for the small ones (like Gazzetta now).

Like with so many things, AI can optimize countless aspects of research work, along with other tactical aspects of publishing like finding usage patterns in digital analytics, optimizing paywalls, analyzing comments for hate speech, and any other information processing at scale within defined contexts.

But here's the danger: mistaking tactical efficiency within systems for strategic innovation of systems.

For instance: If AI helps in our ability to process digital metrics, we’re just getting better at watching a fish swim in an aquarium but we can’t claim this helps us understand ocean ecosystems. (Call it the user needs model fallacy?) You're seeing behavior within a predefined system, not necessarily the broader context or potential.

The more pertinent question isn't whether AI can help us process research faster or find patterns in data. It's whether any of this brings us closer to actually helping people make sense of the world. If AI research leads to more efficient production of irrelevant content, we're just accelerating our own irrelevance.

Better metrics won't fix a broken model

Nick McGreivy, a Princeton physicist, recently wrote about AI use in his field: "AI adoption is exploding among scientists less because it benefits science and more because it benefits the scientists themselves."

It’s easy to mistake these improvements in efficiency as success in their own right. (Yes, I know, but still!)

And there's another wrinkle: researchers (and definitely not journalists) aren't incentivized to publish negative results. We'll probably see lots of successes at conferences, in Slacks and LinkedIn write-ups, but fewer of where AI output misled and led to failure.

Not only are there few incentives to share such learnings, they’re so hard to spot, especially so in research. This is where we risk fooling ourselves. We can use AI to get 30% higher engagement, 50% more reach, 100% more efficiency in production, all while our actual utility to real people may continue its steady overall decline.

We risk being like a restaurant announcing they've optimized their kitchen operations while their customers quietly stop returning and potential new ones never walk through the door.

Some metrics improve. Some output may look impressive. The grant applications may succeed. Meanwhile, the fundamental question - are we helping anyone? - risks going unasked.

Especially as AI intermediates not just our work but also how information is consumed more broadly, we really shouldn’t be using AI to win a game that matters increasingly less.

The blockchain parallel

But even when we commit to transformative exploration, a problem remains: the hype cycle of excitement and fatigue.

Remember the blockchain hype that was especially acute in the media space a few years back?

I was, and remain, interested in decentralized hosting and publishing. I still occasionally get laughed off for seeing some promise in solutions like the InterPlanetary File System, but it’s that hype that overshadowed and ultimately probably also prevented some more cool-headed exploration.

A recent study documented how the hype around blockchain led to unnecessary cost in the humanitarian sector: it's the story of a blockchain pilot in Jordanian refugee camps that promised to transform aid delivery. When examined closely, the technology was unnecessary. A simple database would have sufficed!

The real beneficiaries? Aid organizations enhancing their "innovative" image and securing funding that could have helped more effectively elsewhere.

Sound familiar? Same risk with AI in journalism? So much. The lesson isn't to become either breathless technological optimists or reflexive deniers.

How about a pragmatic approach that asks: Where might these tools genuinely enhance our ability to identify and serve information needs? How can we assess utility beyond technological spectacle?

The more sophisticated our research tools become, the further we risk drifting away from understanding actual human experience by making us assume a deterministic nexus.

We're building telescopes when what we need are dinner tables.

AI as an admission of weakness (and why that's liberating)

I've come to look at AI in strategic audience research as an admission of weakness, including mine.

It's my weakness that years of research went unused. My weakness is that I can't directly access some people I need to understand better. It’s my weakness that I work in the context of a newsroom that has not fully understood its existential relevance challenge.

I believe this explains why many well-intentioned researchers are exploring synthetic personas and other AI-powered research solutions. They're operating within constrained systems, genuinely trying to make insights more accessible and useful to existing contexts of newsrooms that may be resistant to adopting a theory of service but receptive to technological innovation.

Framed as a weakness, AI loses its futuristic luster and is reduced to what it is, at least right now: a mighty tool for challenging situations that require compromise in getting to slightly better strategic decisions.

While we should certainly critique misleading or reductive applications, I also think we should be careful not to dismiss anyone approaching these technologies with genuine curiosity and genuinely good intentions to solve real problems or improve systems.

I think the real question is the intent. If the intent is to become more useful for people, and it doesn't work, then there will be some correction and the validation will happen in the real world.

If the intention is rent-seeking, the opportunistic buzzword bingo will eventually become apparent and there will be little to build on further, much like with the blockchain experiment in Jordan.

Intent as criteria

This is how I landed: AI tools can help optimize existing systems, reduce obstacles of cost and time of analysis.

For anything strategic - deciding what needs to be done, what venture to pursue - it is an admission of weakness that can help if handled with care, but if used wrongly, will lead you down the wrong path.

Underlying that distinction of tactical strength and strategic weakness is my continued belief that the messiness in human experience can't be fully captured.

That's an eternal limitation of all research. It’s also not its point. The point is to give some people greater empathy for other people they seek to serve.

In this, AI-powered research isn’t different from the unused or wrongly-premised traditional research I’ve worked with. This isn’t ever deterministic.

That's what gives me hope that even in future information ecosystems in which AI intermediates so much more information than it already does now, we will be able to preserve some forms of human information sovereignty in all of its messy quirkiness.

Our competitive advantage isn't computational power, but our ability to hold multiple truths simultaneously, our care and sometimes individualistically-irrational solidarity. We feel our way through uncertainty, creating (sometimes irrational) value (and beauty) no algorithm could conceive.

Data points are not destinations. They only optimize the journey.

Especially in information services, surrendering our strategic imagination to mathematical thinking is amputating our capacity to transform information into meaning, care into action as services.

Framing AI as an admission of weakness rather than a solution in research may actually be freeing: It strips away the pressure to evangelize and the performance of innovation. It also frees from debilitating doubt. It introduces healthy skepticism while acknowledging the real constraints we face and maintains a focus on utilitarian progress.

Yes, we can't reach everyone directly, that’s why in this project we have to look elsewhere. Yes, our research methods have limitations, that’s why we need to find other ways. By naming these weaknesses honestly, we can use AI pragmatically.

It's a crutch where needed, not a cure-all to blindly believe in. No one likes a crutch, but it is sometimes helpful.

What if…

What if we used AI not to predict what content will perform, but to surface patterns in information gaps that journalists could then address? What if AI helped us see connections we're missing, like patterns in questions people ask or problems they're trying to solve, that we could then use our human judgment to serve with new or curated information or psychological support?

It would just help us see what we might miss, so we can do the human work of designing meaningful service for other humans.

Questions

If you're using AI in your newsroom, maybe a good question to ask yourself: Is this bringing us closer to people, or giving us an excuse to stay distant?

If you're a funder, perhaps don't reward AI initiatives that promise "newsroom transformation" without defining what genuine service looks like. Fund experiments that use AI to surface information otherwise lost or increase efficiency to free more time to talk to people, but not just optimize existing operational assumptions.

If you're a researcher or experimenting, share your fails! You can set an example. This apology/reflection by the Chicago Sun-Times is an inspiration. I will try to do better in this myself.

Challenge any AI-powered knowledge base with structured queries, guardrails and skepticism to show you what you may be missing, not confirm what you think you know. You’re dealing with a sycophant, not a peer.

We risk using AI to accelerate the irrelevance of the journalism industry in its current form. Instead, we could use it to advance its adaptation to new ecosystems in which journalism, the human craft, is preserved.

But the latter requires admitting that our problem is not really technological. It is forgetting that journalism exists to serve people, not the continued existence of institutions as they currently exist; and that the task of identifying that service and providing it with the craft of journalism is fundamentally ours, human. (It is also just so much more fulfilling way to spend one’s limited time.)

Looking back

Some AI-powered tools I've found useful:

• NotebookLM for contextual document (and used creatively, some interview) analysis;
• Good Tape for straightforward transcription;
• Elicit for academic literature discovery;
• CoLoop for transcription, qualitative review and theme extraction;
• Yazi for reaching participants through WhatsApp with AI-augmented follow-up questions;
• Qurio for embedding research or engagement questions within content;

Each serves a specific, limited but valuable tactical purpose. None can ever replace strategic thinking. I presented some of the work and these tools at the Hacks/Hackers conference in Baltimore (my full slide deck). 

Last week at the Lenfest News Philanthropy Summit in Philadelphia, I ran workshops on board governance (my handout on some board member ideal types I wish someone had shared with me years ago) and impact measurement (handout) with dear friends, Leonor and Madison. Here's an overall write-up of the event. Thank you, Yossi and Diana for having us.

During my preparations, this Rest of World piece came up as an example of what meaningful impact looks like that stuck with me: navigation apps and WhatsApp groups helping people in the West Bank navigate checkpoints. As one chat group manager put it: "This is a collective effort to save lives."

This is civic media stripped to its essence: information tangibly helping people navigate their lives. It's the kind of impact opportunity we could be aspiring to uncover with research: not just reach or engagement sizing, but human, collaborative information provision that genuinely helps people solve real problems.

(If you’re interested in reading more, here are two literature reviews by Elicit on the ethical and methodological challenges of synthetic personas. I also recommend this, this and this.)

Looking ahead

I'll be crossing the Atlantic a few times in the coming weeks with trips to Brussels, Bonn and Zurich. Virtually, I'll be talking about AI in research at the Rosenfeld Designing with AI conference in June (code BOEHLER-DWAI2025 gives you $75 off).

I'll also start working on my presentation for Hackers On Planet Earth (HOPE) in New York, Aug. 15-17, where I'll try to make a case for course correction in the media/internet freedom space, which I worry has become somewhat self-referential.

I want to pitch what collaboration between technologists and journalists focused on creating information services that actually help people navigate their lives could look like. If you're interested in these questions or planning to attend, let me know.

No free coaching slots this month - too much travel and some project deadlines are looming. They'll be back next month, I promise.

But conversation is how I think best. If anything here sparked agreement, disagreement, or that feeling of "yes, but…", I'd love to hear it. Hit reply or message me on Signal at patrickb.01. I'm grateful for your thoughts.

That familiar dread I mentioned? It's not about AI itself. It's about watching our industry once again mistaking efficiency for effectiveness, activity for impact. We've done it with metrics, with platforms, some even with blockchain. Each time, we've accelerated our irrelevance while congratulating ourselves on our innovation.

Maybe this time we can choose differently. Maybe we can use these tools to get closer to people, not further away. Maybe we can finally admit that our problem was never technological - it was forgetting what journalism is for and how it can matter in the first place.

Until next month!