Simulating Human Survey Responses for Market Research

Background

Survey-based market and user research is both essential and expensive. Recruiting participants, conducting interviews, running diary studies, and administering questionnaires across multiple rounds demands significant time and resources. This cost creates pressure to find computational approaches that can extend or support traditional methods.

Large language models (LLMs) have attracted growing attention in this context because they can produce coherent, contextually plausible responses to survey-style questions. The practical question for market research is whether these responses can go beyond generic plausibility and actually reflect the preferences and reasoning patterns of specific individuals. Several studies show that LLMs can approximate aggregate trends [2, 4], but whether they can simulate individual behavior with meaningful fidelity is a different and harder question.

Without grounding in participant-specific data, an LLM response may appear convincing while drawing primarily on statistical patterns from training rather than from the individual being modeled [3, 7]. This study investigates what happens when you systematically increase the amount and richness of personal context provided to an LLM agent, and whether doing so produces measurably better alignment with real human survey responses.

Study Design

We recruited 13 participants based in Germany and collected data over six weeks using a multi-method approach. This included two in-depth interviews per participant (each 90 to 120 minutes), covering everyday routines, purchasing behavior, attitudes, and financial habits; a six-day audio diary study capturing spontaneous daily reflections; a structured online survey comprising 111 questions; and a final round of evaluation surveys spanning four market research domains. All audio data was transcribed using Whisper [6], anonymized, and processed within the European Economic Area.

This dataset formed the shared foundation for building and comparing eight distinct LLM agent variants. Each variant received a different type and quantity of participant information during response generation, allowing us to isolate the effect of contextualization under otherwise identical conditions.

Agent Variants

The eight configurations are organized as a progressive sequence, from no context to full integration. At the baseline, the Random agent answers questions with no framing whatsoever. The Generic agent is instructed to respond as a human survey participant, relying on general language priors. Subsequent variants introduce participant-specific context in increasing depth: basic demographics (either human-curated or AI-extracted), rich persona profiles including psychographic and attitudinal attributes, retrieval-augmented generation (RAG) drawing directly on raw interview and diary excerpts, psychological reflections built around the Big Five traits and behavioral patterns, and finally a Fusion configuration combining all layers through chain-of-thought reasoning and multi-query retrieval.

Results

We evaluated 1,246 response pairs using OpenAI's O3 model as an LLM judge, assigning similarity scores from 0.0 to 1.0. These scores were triangulated with BERTScore, Jaccard similarity, and Spearman correlation to ensure robustness across different evaluation signals.

Agent	Context provided	Similarity score
Fusion	All layers combined	0.78
RAG	Retrieved raw excerpts	0.76
Persona	Demographics + psychographics	0.75
Reflections	Psychological profiling	0.74
Demo-H	Human-curated demographics	0.74
Demo-AI	AI-extracted demographics	0.72
Generic	Task framing only	0.68
Random	None	0.60

All contextualized variants significantly outperformed the Generic baseline, confirmed by Welch's t-test with Benjamini-Hochberg FDR correction and cross-checked with the Mann-Whitney U test. The Fusion configuration achieved the largest improvement over baseline (mean difference +0.092, Cohen's d = 0.336, p < 10^-15). Effect sizes were moderate across all contextualized variants, reflecting a pattern of incremental rather than transformational gains.

Progressively richer contextualization is associated with stronger alignment with human responses, though the gains are incremental and come with increasing data collection and engineering complexity.

Key Findings

Even simple task framing (Generic, 0.68) substantially outperforms no framing at all (Random, 0.60), because the model can infer cultural and linguistic cues from the survey questions themselves. This means the first layer of contextualization, correctly framing the model's role, carries disproportionate weight relative to its cost.

Human-curated demographics (Demo-H: 0.74) outperformed AI-extracted demographics (Demo-AI: 0.72), suggesting that the source and precision of demographic data influences alignment, not just its presence. Persona profiles extended this further by capturing attitudinal and motivational dimensions alongside factual attributes. The RAG configuration performed strongly by grounding responses in the participant's actual words, though this requires the raw corpus to remain available at inference time. The Fusion agent, which combines all layers through automated retrieval and chain-of-thought reasoning, achieved the highest and most stable performance across survey items.

Methodological note: Given the sample size (n = 13), these results should be interpreted as exploratory rather than confirmatory. Confidence intervals are reported for all mean differences. The Fusion configuration evaluates the integrated effect of all context layers; ablation of individual components is a direction for future work.

Implications

These findings position contextualization as a design decision with real trade-offs, not simply a technical optimization. The data required to achieve the highest-performing configurations, multi-session interviews, longitudinal diaries, structured surveys, closely resembles the research effort that AI simulation is intended to reduce. Practitioners must therefore weigh the cost of data collection against the fidelity gains it enables, and consider whether the use case demands high-fidelity individual simulation or whether a demographic-grounded agent producing scores in the 0.72 to 0.74 range is sufficient.

The study also raises questions that go beyond performance metrics. Agents grounded in extensive personal data risk flattening individual nuance into a statistical composite, or reproducing demographic stereotypes rather than authentic individual perspectives [5, 7]. As the practice of LLM-based human simulation matures, questions of consent, transparency, and the conditions under which synthetic agents add genuine value without compromising authenticity will require sustained attention.

---

References

1. Saini, B.K., Kumar, C., Pollmann, K., Bierkandt, J., Janssen, D., Knecht, C., & Fronemann, N. (2026). Simulation of Human Survey Responses for Market Research: The Role of Contextualization in LLM-Based Agents. CHI EA 2026. https://doi.org/10.1145/3772363.3798991
2. Brand, J., Israeli, A., & Ngwe, D. (2023). Using LLMs for Market Research. Harvard Business School Marketing Unit Working Paper 23-062.
3. Abeliuk, A., Gaete, V., & Bro, N. (2025). Fairness in LLM-Generated Surveys. arXiv:2501.15351
4. Arora, N., Chakraborty, I., & Nishimura, Y. (2025). AI-human hybrids for marketing research. Journal of Marketing, 89(2), 43-70.
5. Li, A., Chen, H., Namkoong, H., & Peng, T. (2025). LLM Generated Persona is a Promise with a Catch. arXiv:2503.16527
6. Radford, A., Kim, J.W., Xu, T., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356
7. Wang, A., Morgenstern, J., & Dickerson, J.P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3), 400-411.
8. Park, J.S., Zou, C.Q., Shaw, A., et al. (2024). Generative Agent Simulations of 1,000 People. arXiv:2411.10109

Sources

Read on Medium

Related on the shelf

• ProjectSozioMimic