The Ghost in the Gallery
Why digital twins aren't a society yet
Imagine a gallery filled with a thousand portrait paintings, each one an uncanny likeness of a real person. The detail is extraordinary. The eyes seem to follow you. In another room, those same portraits have been given voices and memories. They can answer questions. They fill out surveys. They even predict what the real person might buy next.
This is roughly where the field now stands. In our own work at Fraunhofer IAO, we examined how LLM-based agents can simulate human survey responses for market research, finding that richer personal context brings the simulation meaningfully closer to what real respondents would say [1]. The richer the context given to an agent, the more it begins to sound like a person rather than a demographic category.
Several other research groups have reached similar conclusions, each pushing individual fidelity a little further [2]. The technical momentum is real. But a question keeps coming back to me as I sit with this work: if we put a thousand of these well-constructed twins in a room, would we actually have a society?
Intelligence is not a solo act
We tend to measure intelligence as a set of individual capabilities: memory, reasoning, prediction, task completion. By those measures, a well-constructed digital twin can already outperform its human original in several respects. But that framing was always too narrow.
Edwin Hutchins, in his study of how a naval ship navigates, showed that cognition is not located inside any one person's skull. It is spread across people, tools, gestures, diagrams, and shared histories of practice [3]. A cockpit, he argued, does not merely assist the pilot who sits in it. It is itself a thinking system. The intelligence is distributed. Remove one piece and the whole computation degrades.
True intelligence lives in the spaces between us. It is found in the friction of a disagreement, in the way a negotiation shifts the goals of both parties, and in the trust two people build without ever naming it.
Andy Clark extended this line of thought, arguing that the human mind routinely offloads its work onto the environment and onto other people [4]. What we call individual cognition is better understood as a partnership between a brain and its surroundings, including the other brains it is in contact with.
A digital twin, however faithful to a single person, is designed as a closed system. It has no surroundings. It has no friction. It cannot be genuinely surprised by another mind, because nothing in its architecture requires it to update on the shock of real disagreement.
What our market research work revealed
Working with real consumer survey data, our team at Fraunhofer IAO tested what happens when you vary how much personal context you give an LLM agent before asking it to stand in for a respondent [1]. A thin persona, built only from demographic variables, produces answers that are plausible but flat. Agents with richer context, ones that know something about the person's past choices, values, and environment, get closer to individual-level truth.
But here is the thing we kept noticing at the edges of the data: even a well-contextualized agent answers each question in isolation. It does not carry the tiredness from question seventeen into question eighteen. It does not quietly change its mind because of something that came up earlier in the session. Other researchers have noted similarly: LLMs can reproduce mean responses well, but tend to be systematically overconfident and display artificially low variance across a population [5]. The survey instrument, in the hands of a digital twin, becomes frictionless. And frictionless is not the same as accurate.
The problem with automated teams
Consider the analogy of a software testing team replaced entirely by agents. The agents would be faster, more consistent, and tireless. They would find more bugs per hour than any human crew. But here is what gets quietly traded away: the conversation beside the coffee machine where a developer says, without knowing it matters, that something about the error pattern feels like something they saw on a different project three years ago. The insight that arrives sideways, from a mind whose attention was elsewhere.
Researchers studying multi-agent AI systems have found that getting agents to coordinate is, technically, a solved problem. What is far harder is getting them to generate meaning together. Current LLM-based multi-agent systems can reproduce believable social behaviour in simulation, but they must be given social norms from outside. They do not generate those norms from lived conflict [6].
In human groups, norms do not arrive pre-installed. They emerge from disputes about what was fair, from the slow embarrassment of having said something that landed badly, from the memory of what a group survived together. That residue is not data. It is something more like scar tissue, and it shapes every future decision.
The emergence gap
In complexity science, emergence refers to properties that arise from the interaction of parts but cannot be predicted or reduced to any single part. Water is wet. A hydrogen atom is not. Wetness is an emergent property of the interaction of many molecules under the right conditions.
Human society produces emergent meaning in a similar way. Values shift through public argument. Language mutates through contact between groups who do not yet share a vocabulary. Identity itself is partly forged in the space where one person's self-story meets another's refusal to accept it. None of this can happen inside a single agent, no matter how richly personalised.
Research published at IJCAI 2024 proposes architectures for norm emergence in multi-agent systems, describing a cycle of norm creation, spreading, evaluation, and compliance [7]. It is careful, ingenious engineering. But the authors note that LLMs do not adequately understand culture-specific norms, especially when trained on text from diverse backgrounds with unequal representation. The agents can spread a norm. They cannot yet feel its weight.
Matching a human's IQ is a technical milestone. Matching a society's collective wisdom is a different mountain entirely.
This is what I call the emergence gap: the distance between coordinated performance and co-created meaning. Hong and Page (2004) showed formally that groups of diverse problem-solvers outperform groups of high-ability individuals on hard problems [8]. The diversity is not incidental. The cognitive friction between people who see things differently is, itself, the mechanism that generates better solutions. A gallery of a thousand perfect portraits, each faithfully individuated, does not automatically produce that friction. It may in fact eliminate it.
What this means for how we design
Digital twins as currently constructed are impressive instruments for studying individual behaviour. Work assembling datasets of thousands of twins across hundreds of question domains shows how far individual fidelity can be pushed [9]. In the right context, this accuracy is genuinely useful for market research, policy pre-testing, and social science at scale.
But the limit appears when we ask what a collection of such twins can tell us about how meaning evolves in a group over time. Even our own findings at Fraunhofer IAO suggest that contextual richness improves individual simulation significantly, but the gap that remains is not a calibration problem. It is a structural one. You cannot simulate the evolution of shared meaning by running many individual simulations in parallel and averaging the outputs.
The LLM that powers each twin was, in a sense, trained on the traces of human emergence. It absorbed the outputs of centuries of collective sense-making. What it did not absorb is the process. You cannot learn to negotiate by reading the records of past negotiations. You have to be in the room.
If we want AI systems that can genuinely contribute to collective intelligence rather than merely simulate isolated individuals, the design challenge shifts. It is no longer a question of making a single agent more accurate. It is a question of what kind of architecture allows new meaning to form in the encounter between agents, human and artificial alike, in real time, under conditions of genuine uncertainty.
That is harder. It requires us to think less about the fidelity of each portrait, and more about what happens in the space between them.
A living crowd is not a gallery. It is what happens when the portraits begin to argue.
References
- Saini, B. K., Kumar, C., Pollmann, K., & Fronemann, N. (2026). Simulation of human survey responses for market research: The role of contextualization in LLM-based agents. CHI EA 2026. Fraunhofer Institute for Industrial Engineering IAO. https://doi.org/10.1145/3772363.3798991
- Park, J. S., et al. (2024). Generative agent simulations of 1,000 people. arXiv:2411.10109. https://arxiv.org/abs/2411.10109
- Hutchins, E. (1995). Cognition in the Wild. MIT Press. Also: Hutchins, E. (1995). How a cockpit remembers its speeds. Cognitive Science, 19, 265-288.
- Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7-19.
- Bisbee, J., et al. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis.
- Li, S., et al. (2024). Spontaneous emergence of agent individuality through social interactions in LLM-based communities. Frontiers in Psychology. PMC11675631.
- Liu, C., et al. (2024). Emergence of social norms in generative agent societies: Principles and architecture. Proc. IJCAI 2024, 7895-7903. https://doi.org/10.24963/ijcai.2024/874
- Hong, L., & Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. PNAS, 101(46), 16385-16389.
- DAPLab, Columbia University (2025). Twin-2K-500: A dataset for building digital twins of over 2,000 people. Marketing Science. https://daplab.cs.columbia.edu