AI "Participants" as Synthetic Users

1. Introduction: The Rise and Controversy of AI Synthetic Users 🤖

• A flood of AI-based tools has hit the user and product research space, but few have stirred as much controversy as AI-generated users — synthetic users.
• What are synthetic users?

  "Synthetic users are AI-generated profiles that attempt to mimic a user population, providing artificially produced research findings without studying real users."

• They can simulate and express thoughts, needs, and experiences based on training data through a chat interface.
• The core idea:

  "By interviewing AI-powered participants, you can learn about the needs and behaviors of a user population without speaking to real people."

• The presenter, as a researcher, candidly admits this approach "largely sounds terrible," but also understands its appeal.

  "Recruiting real people, scheduling, and having conversations takes time, money, effort, and energy. You don't always have the luxury. Just talking to AI users can sound pretty tempting to a lot of people."

2. The Experiment: Synthetic Users vs. Real Users 🧑‍💻

• To investigate reliability and usefulness, an experiment was conducted using two tools:
  • ChatGPT and
  • Synthetic Users, a dedicated AI user platform.
• Results from real human research were compared against AI user interview results.

3. The Limitations of Synthetic Users 🚫

• The biggest problem:

  "AI users tended to be far more positive than real humans."

• A characteristic of AI chatbots:

  "AI chatbots have a tendency to please — sycophancy — and don't always model human behavior well."

• Idealized and unrealistic answers:

  "When asked about their everyday behavior, synthetic users always gave clean, ideal, and unrealistic answers."

• Example:

  "When asked whether they completed all their online courses as planned, synthetic users always said they finished every course on schedule."

• Real humans are different:

  "Humans aren't like that. We're complex. In the real world, we don't always behave perfectly and predictably." "Real users in our research said they frequently didn't finish online courses they'd started, for reasons like being too busy or losing interest."

• The essence of UX research:

  "UX research explores human behavior. To understand why people make the choices they do, what contextual factors influence behavior, and what matters and when — you need to hear from real humans." "These are things you simply cannot learn from today's synthetic users."

4. Where Synthetic Users Can Be Useful 📝

• They're not entirely useless!

  "Synthetic users are an excellent tool for desk research."

• Useful for learning what's already known:

  "They're well suited for learning what is already known about a particular user group."

• Helpful for preparing to research unfamiliar domains:

  "When you're about to research users in a domain you're not familiar with, running a mock interview with synthetic users can give you insight into how to structure and conduct the real research."

• But what about the information that actually gives you a competitive edge?

  "If you want accurate, novel information about your users — information that puts your company ahead of its competitors — that can only come from real humans."

5. Looking Ahead: Research-Based Synthetic Users 🔮

• Even at the time of filming:

  "There are companies developing the next version of this concept."

• Research-grounded synthetic users:

  "Instead of drawing from random information on the internet, imagine synthetic users built on your own actual research findings — research-based personas you can actually have a conversation with."

• Something to watch closely:

  "As AI tools for research continue to proliferate, this kind of development is worth keeping a close eye on."

6. Closing and Additional Resources 🎬

• A note of thanks and further guidance:

  "Thank you for watching. If you'd like to see more UX videos, check out the ones here, and consider subscribing to the channel." "At nngroup.com, you can access a library of over 2,000 free articles, and you can also enroll in UX courses offering live, hands-on UX training."

Key Keyword Summary

• Synthetic Users
• AI-based user research
• Real users vs. synthetic users
• Sycophancy
• Idealized / unrealistic answers
• Desk research
• Research-based personas
• The essence of UX research: exploring human behavior

The Lines to Remember from This Video

"AI users tended to be far more positive than real humans." "Humans aren't like that. We're complex." "UX research explores human behavior. These are things you cannot learn from today's synthetic users." "If you want accurate, novel information about your users, that can only come from real humans." "If you could have a conversation with a research-based persona, that would be a truly noteworthy development."

We hope this summary helps you quickly grasp the current state, limitations, and future potential of synthetic users and AI research! 😊