This paper tests whether persona-conditioned LLM agents, built from real users' anonymized behavioral data, can predict which way an A/B test will go without using any real traffic. Across 40 e-commerce A/B tests, pairwise rating and domain-matched behavioral data proved especially important, and the approach reached 0.75–0.90 directional accuracy depending on the setup. The conclusion, however, is that it is better used not as a replacement for real experiments but as a low-cost pre-screening tool that filters out weak candidates in advance and sets priorities.

1. The Cost of Real A/B Tests and the Research Goal

Online A/B testing is the most reliable way to confirm whether a product change actually works. But it requires enough user traffic, engineering work to implement the experiment, and usually several weeks before a statistically meaningful result arrives. As a result, companies struggle to test many ideas at once and end up spending money and traffic even on designs that are likely to fail.

To reduce this problem, the paper proposes having a large language model (LLM) play different user roles to simulate reactions to an A/B test in advance. The key point is not simply to invent virtual consumers, but to build personas from anonymized behavioral signals accumulated over several months. These include activity frequency, purchase and browsing patterns, engagement level, price sensitivity, and inferred demographic traits.

"Data-driven personas are grounded in real behavioral data and are not meant to represent or re-identify any specific individual."

Among prior work, SimAB generated synthetic personas from screenshots to predict the direction of A/B tests, but it was not grounded in actual user behavior. SimGym, by contrast, replays real traffic logs through browser agents, but it requires detailed session logs and does not go deeply into how the personas themselves should be constructed.

This study targets the gap between the two. It systematically examines how useful personas built from real behavioral data are, how deep a persona's information needs to be, how well it must cover diverse user groups, and whether performance holds up when the number of personas is reduced. 🎯

2. Core Research Questions and Contributions

The researchers built a benchmark of 40 e-commerce A/B tests spanning two kinds of metrics, click-through rate (CTR) and subscription conversion, and examined the following four questions.

1. How should the agent be asked in order to improve accuracy?
2. Are personas based on real behavioral data better than synthetic personas?
3. What balance is needed between capturing each person's behavioral history in depth and covering a more diverse population broadly?
4. How far can the full persona pool be reduced while preserving simulation quality?

The paper's key contributions can be summarized as follows.

• It defines A/B test simulation as a structured questioning task and systematically compares question formats.
• It shows that domain alignment, meaning how well the domain of the behavioral data matches the subject of the experiment, matters far more than the sheer amount of persona data.
• It empirically analyzes the depth–diversity trade-off between the information density of individual personas and the representativeness of the overall pool.
• It finds that cutting the number of personas roughly in half causes almost no quality loss, pointing to inference cost savings of up to 2x.

3. The A/B Test Simulation Framework

The framework consists of three main stages: persona construction, LLM simulation, and aggregation and evaluation.

Persona Construction

First, anonymized and aggregated user behavior records are fed to an LLM. From this data, the LLM generates consumer profiles in a consistent format. These include not only inferred demographic information such as age, gender, income, and education level, but also brand preference, price sensitivity, quality expectations, how the user relies on reviews, purchase frequency, spending level, the ratio of product views after searches, and the ratio of purchases after views.

The behavioral data is also turned into a readable narrative profile. For example, it is organized into a coherent story such as "a consumer who frequently compares low-priced products and checks reviews carefully," so that the LLM can later act out a specific user role.

"You will now embody the persona of a specific online retail customer. Respond in line with this person's demographic background, interests, and behavioral patterns."

Importantly, these personas are not copies of real individuals. They are synthetic profiles that statistically reflect real behavioral patterns, and the researchers state explicitly that they are not used for identifying or targeting individuals.

Simulation

For each A/B test, the persona agent is shown the visual screens of the existing design, the Control, and the modified version, the Treatment. The agent evaluates both designs from the perspective of its assigned persona and returns a score and reasoning related to the target metric.

For a CTR experiment, for example, it is asked, "Does this screen make you want to click?" and for a subscription experiment, "Does this screen increase your intent to sign up?" The differences between the scores each agent gives to Control and Treatment are then aggregated to predict whether, in the real A/B test, the treatment would have performed better, worse, or had almost no effect.

Aggregation and Evaluation

The goal of the study is less to pin down the exact effect size than to get right whether the treatment effect is positive or negative, in other words the direction that matters for the real product decision: "Is this worth shipping?"

To do this, three metrics were used.

• Accuracy: whether the real result and the simulation predicted the same direction.
• Sign Overlap: compares how confident both are that the effect is positive, or how confident both are that it is negative.
• Sign Bhattacharyya: evaluates how similar the two predictions are, taking their uncertainty distributions into account.

With simple accuracy, when an effect is close to zero, even a tiny difference can flip the result sharply between right and wrong. The paper therefore also used the two distribution-based metrics, which reflect confidence levels and uncertainty.

4. How Question Format Determines Accuracy

The researchers expected that how designs are presented to the agent and how answers are collected would strongly affect performance, and compared four formats.

• Independent binary: show each design separately and ask for a yes/no answer on whether the persona would engage.
• Independent rating: show each design separately and ask for a score from 1 to 10.
• Pairwise binary: show both designs together and ask for a yes/no answer on whether the persona would engage with each design.
• Pairwise rating: show both designs together and ask for a score from 1 to 10 for each.

In the pairwise formats, the presentation order was randomized for each persona to reduce position bias, where the left/right or first/second order of the designs affects the outcome.

The results showed that pairwise rating, which shows both designs together and asks for scores, was the most consistently strong across all metrics. The independent formats performed poorly because the agent struggled to assign absolute scores consistently. Pairwise comparison, by contrast, provided a reference point, "which of the two is better?", which helped the LLM make comparative judgments.

"Independent formats performed poorly because scores are hard to calibrate without comparative context, while pairwise comparison provided a natural reference point."

Pairwise rating recorded directional accuracy of 0.75 on CTR and 0.80 on subscription tests. Pairwise binary was also strong on the subscription task but weaker on CTR. All subsequent main experiments therefore used pairwise rating as the default setting.

5. Data Sources and Why Domain Alignment Matters

In addition to personas built from on-platform behavioral data, the researchers compared personas built from three public datasets. The public data consisted of social science survey data, movie rating data, and public e-commerce purchase data.

Personas based on platform behavioral data achieved 0.70 accuracy on CTR and competitive performance on subscriptions as well. But personas built from public e-commerce data recorded 0.90 accuracy on subscription tests, actually outperforming the platform-specific personas at 0.80.

This means that whether the task and the data belong to the same behavioral domain matters more than whether the data is proprietary or internal to the platform. E-commerce browsing and purchasing behavior is directly relevant to evaluating widget clicks and subscription intent, so even public data could provide a strong enough signal.

By contrast, the Rotten Tomatoes personas, based on movie ratings and entertainment tastes, worked to some extent on CTR at 0.65 but dropped to 0.60 on subscriptions. Personas based on social science surveys also stayed at a middling level on both tasks.

"Domain alignment is central to persona effectiveness. Even public e-commerce data can match platform-specific personas."

In other words, you should use behavioral data that fits what is being evaluated: shopping behavior data if the experiment concerns a shopping experience, and content consumption history if you are testing content preferences.

6. Balancing Behavioral Depth and Population Diversity

The researchers compared two persona pools of 935 personas each.

The Deep pool was built around users with rich activity levels and purchase histories. Each persona has more behavioral information, making it possible to build detailed, confident profiles. However, it is skewed toward active power users and risks not representing the overall user population well enough.

The Representative pool, by contrast, secured diversity by drawing stratified samples from multiple demographic and behavioral groups. Individual users' behavioral records are sparser, but overall it includes groups that are scarce in the deep pool, such as younger users, renters, and users without a college degree.

On CTR, the deep pool was significantly better than the representative pool, with accuracy of 0.75 versus 0.60. This is interpreted as a sign that, for personas with sparse behavioral histories, the LLM may fall back on answers close to general common sense rather than making judgments grounded in individual traits.

On subscription tests, however, both pools recorded 0.80 accuracy, and there was no significant difference on the other evaluation metrics either. This suggests that even when individual profiles lack depth, including diverse population groups can compensate at the level of the pool's overall predictive power.

Both pools, however, still had demographic composition biases. The representative pool was concentrated in people in their 30s and 40s with few older users, while the deep pool had a relatively wider age spread but lacked younger users. The researchers noted that this may not be simply a sampling problem, but a bias arising as the LLM infers demographic traits from behavioral data.

"Behavioral depth gave a significant advantage in CTR accuracy, but on subscription tests demographic diversity offset the difference."

7. Cutting Costs by Reducing the Number of Personas

As the number of personas grows, LLM call costs and processing time increase linearly. The researchers examined whether performance could be maintained by selecting only 500 personas from the full pool of 935.

They compared three methods.

• Random sampling: select uniformly at random from the whole pool.
• Greedy Farthest: sequentially select the persona farthest from those already chosen, securing diversity in the feature space.
• Kernel Herding: choose a subset whose distribution resembles that of the full population as closely as possible.

These methods operate in an embedding space where the persona texts are converted into vectors. In other words, they avoid repeatedly picking personas that are similar in content and instead select so as to preserve the full range of traits in the overall population.

When 500 personas were selected, every method maintained nearly the same performance as the full pool. On CTR, the gap from the full pool was within 1 percentage point, and on subscriptions some results were equal or even better. Greedy Farthest was best on CTR and Kernel Herding on subscriptions, but simple random sampling was also quite strong.

In some cases, the subsets even scored better than the full population on uncertainty. Using every persona can produce overconfident predictions, whereas a smaller sample introduces natural variability that can better match the uncertainty of the real experiment.

Even a small sample of just 100 personas produced competitive results, with 0.77–0.80 on CTR and 0.82–0.86 on subscriptions.

"Even reduced to 500, the pool maintained nearly full-pool accuracy, showing potential cost savings of up to 2x."

8. Component Contributions and Cross-Model Validation

To find out which elements actually contributed to the performance gains, the researchers ran ablation experiments.

First, personas using only demographic information were much weaker than those using full behavioral profiles. Subscription accuracy in particular fell from 0.80 with full behavioral personas to 0.30 with demographics alone. This means that for more deliberate or important decisions such as subscribing, actual behavioral history and purchase tendencies matter far more than basic attributes like age, income, and education.

In addition, when agents were made to output only a score without explaining their reasoning, CTR accuracy dropped from 0.70 to 0.60. Subscription accuracy held at 0.80, but the researchers interpreted this as showing that, when predicting relatively subtle click intent as in CTR, prompting the agent to reason step by step based on its persona helps.

Applying a single fixed consumer persona to every test was also ineffective. CTR accuracy was 0.40, lower than the plain LLM's 0.45, and subscription accuracy was 0.60, lower than 0.65.

"A single generic shopper persona did no better than a model with no persona at all."

This shows that simply instructing the model to "act like a consumer" is not enough; many personas reflecting diverse user groups are the key.

In addition to Claude Sonnet 4.5, the researchers repeated the question-format experiment on Claude Haiku and Claude Opus. Which of pairwise rating or pairwise binary did better varied slightly by model, but in all three models pairwise comparison consistently outperformed the independent formats. The conclusion that "comparative context must be provided" therefore does not appear to be limited to a single model.

9. Applications, Trust Boundaries, and Reproducibility

The researchers state clearly that this system cannot fully replace real A/B tests. In practice, it is more realistic to use it as a pre-screening tool that rules out clearly weak candidates in advance and ranks multiple design variants by expected effect. With batch inference, the simulation can finish within a few hours, far shorter than a real experiment that takes weeks. ⚡

In particular, the larger the real effect, the more trustworthy the simulation's predictions were. Conversely, when the effect was close to zero, reliability was low, because even a tiny difference could flip the direction of the result. These results are therefore better seen as a prioritization signal for deciding "which experiment to validate first" than as a binary verdict like "this design must ship."

On the data side, performance dropped noticeably for personas of users with fewer than about 20 transaction records. When behavioral evidence is lacking, the LLM tends to rely on general reasoning instead of personalized judgment. On the other hand, since public e-commerce data performed competitively with platform-internal data, an important advantage is that research can begin even without internal company logs.

Although the researchers cannot release their proprietary A/B test results, they suggest the work can be reproduced by converting public datasets with real preference rankings, such as movie, book, or joke ratings, into an A/B test format. For example, you can treat the difference in average ratings between two movies as the true effect and have persona agents evaluate both movies, creating a directional prediction benchmark similar to an A/B test. On the public datasets, they also confirmed a pattern in which accuracy rises steadily as effect size grows.

10. Limitations and Ethical Considerations

This study has several important limitations. First, the agents see isolated screenshots, not full web pages or real purchase journeys. Real users' decisions are shaped by many contextual factors, such as their current browsing goal, previous sessions, surrounding page content, product prices, and shipping terms, but these were not sufficiently reflected.

The experiments were also limited to 40 tests in a single e-commerce domain and two metrics, CTR and subscriptions. It is not yet known whether the same performance would hold in other industries, multi-step user journeys, or metrics such as long-term retention or actual purchase amount.

Demographic information was not provided directly by users; it was inferred by the LLM from behavioral signals. It may therefore differ from real characteristics, and inference results may vary depending on how particular cues are presented. LLMs can also show biases such as rating the treatment overly positively or being influenced by the item presented first. Pairwise comparison partly cancels out positivity bias that applies equally to both variants, but it is not a complete solution.

Model generalization is also limited. The study mainly used Sonnet 4.5 and partially revalidated on Haiku and Opus, but it has not been confirmed whether the same results would hold for models from other companies or models with entirely different architectures.

On ethics, the researchers state that all behavioral data was anonymized and de-identified in advance, and that no personally identifiable information was included in the pipeline. They also emphasize that persona information such as age, gender, and income is only a statistical conditioning variable for population-level simulation, and is neither treated as an attribute of a real individual nor used for individual profiling or targeting.

11. Conclusion

This paper shows the potential that giving an LLM personas grounded in real behavioral data allows it to predict the direction of A/B test outcomes with reasonable accuracy. The most effective setup was the pairwise rating question format, which shows both designs together and asks for a score for each, with accuracy reaching 0.75–0.90 depending on the task.

The most important lesson is that domain alignment matters more than the quantity or exclusivity of the data. Shopping behavior data was effective for shopping-related A/B tests, and public e-commerce data could match or even beat platform-internal data. Personas with rich behavioral histories had an edge in CTR prediction, but in subscription prediction, covering diverse population groups alone could largely make up the difference.

Ultimately, this approach is less a technology for eliminating real experiments than a tool for picking the more promising experiments under limited traffic and time. If it is combined in the future with more diverse domains, real page context, more sophisticated bias correction, and public benchmarks, it has the potential to evolve into a simulation system that helps decisions before the A/B test stage quickly and cheaply.