SocialSimulation
Agentic orchestration and long-horizon tool-use evaluation
96% of companies surveyed across more than 260 organisations run a form of beta-testing, where customers are given early access to a product to test it before an official launch. Part of the strength of beta-testing comes from the diversity of the customers; in work by senior researchers Margaret Burnett and Anita Sarma, viewing the same product through different cognitive frames surfaced inclusivity issues in over half of the interactions evaluated, more than 90% of which were later confirmed against independent empirical data. As more and more companies try to leverage AI agents to simulate this beta-testing, they struggle to emulate unique personalities and therefore cannot evaluate products effectively.
Social Simulation addresses this problem in three ways: finding novel ways to simulate realistic personalities, incorporating new cognitive perspectives that haven’t been considered, and visualising the process in intuitive ways. This includes using real, existing personas like actors and hosts instead of text-based prompts, and experimenting with ideating unique personas like impatience and radicalism.
The methods developed in this research have the potential to:
- Simulate beta-testing with larger scale, diversity, and accuracy than current methods can achieve.
- Create novel visualisations that make agentic beta-testing intuitive to all clients, especially considering the modern atmosphere of non-technical developers.
- Discover unique friction points in applications by leveraging unexplored cognitive perspectives.
Literature Review
Works Cited
- Centercode. “Beta Testing Industry Survey Report 2016.” Centercode, 2016. Survey of 338 respondents across more than 260 companies; 96% run customer validation tests before release. centercode.com.
- Hilderbrand, Claudia, et al. “Engineering Gender-Inclusivity into Software: Ten Teams’ Tales from the Trenches.” Proceedings of the 42nd International Conference on Software Engineering, Association for Computing Machinery, 2020. PDF.
- Park, Joon Sung, et al. “Generative Agents: Interactive Simulacra of Human Behavior.” Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, Association for Computing Machinery, 2023. PDF.
- Park, Joon Sung, et al. “LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals.” arXiv, 2024. arxiv.org/abs/2411.10109.