Synthetic Research Part II: A Comparison
Part I of this two-part series offered a quick introduction to synthetic research, which is the practice of creating fictional audience members using LLMs to provide a snapshot of how a real audience might respond to a research study question or set of questions. These fictional research participants can take your surveys or provide input inside a focus group. How? Well, by asking your LLM of choice to create personas and take the surveys or answer focus group questions. (Or, you can create the personas -- but isn't it just easier and faster to ask the LLM to do it? …And that’s how our cognitive abilities backslide!)
Part II looks back at a different approach to using AI for research; a collaborative AI tool we designed at my startup Thicket in 2015. At the time, we called the character designer dashboard. It was in many ways the exact opposite approach to synthetic research: We created a real audience-driven fiction designer. Instead of designing fictional audiences to generate answers, we asked real audiences about fictional stories -- in our case, TV shows and movies. We then created a data-driven dashboard with a synthesizer GUI that anyone could use to create characters, storylines, and casting choices, and see how audiences might respond to them. Our clients would use it to create new TV shows and movies that could meet the tastes of real people. And because of our predictive analytics, we could even anticipate how audience's tastes would change over time, so our clients could meet them where they were at.
One of the quizzes that powered our audience insight-powered character design tool.
At the time, I found myself wondering if our tool was dangerous, and if storytellers would be disappointed if they were forced to use a tool that would shut down their desire to create wholly original characters, untethered to the real world. Should they be forced to cater so precisely to what real people want? Would it undermine the quality of the storytelling?
Today, I know that the people who were the most interested and bought our services were ones dedicated to serving underserved audiences. They knew the value of being able to hear from millions of real people, to create inspiring characters and storylines that would have a positive impact on society, making it more diverse, equitable, and inclusive. They were hungry for data that would show that their instincts about what their audiences wanted were right. Yes, it was a tool that appealed to adherents to the (apparently) dreaded values of DEI, but also to those who believed in the value of research.
Let’s be completely clear: Synthetic research is not research. Research gathers real, empirical evidence. Another name for synthetic research is falsified results. The environment that shut down my startup is the same one that has actively promoted the dismantling of a flourishing economy in favor of an oligopolist's paradise, in which the most powerful companies don't have to meet the wants or even the needs of consumers -- they just need power and influence to force people to accept what they're selling. So, I have few doubts about how synthetic research is likely to be deployed and what it’s making possible.
We’re already seeing that Gen AI tools are highly resource-intensive and dangerous for mental health — and over the last decade, I have seen firsthand the lack of care in fixing core issues in the industry. Honestly, I think that if a research outfit decides never to adopt synthetic research, they will be rejecting false science. For this reason, all TLC research services are scoped without the use of synthetic research methods, meaning all participants are real individuals.
So for me, the bottom line is, the current generation of generative AI tools are a completely wretched take on how to make information widely available for all the ways in which people use it professionally. They are not transparent, inclusive, reliable, or accurate, making the data that comes from these platforms not strong enough to support synthetic research purposes in relation to social science research, including the subdisciplines of market or user research.
