Publications

2024

1. NAACL, ICLR’W

   

   Simulating Opinion Dynamics with Networks of LLM-based Agents

   Yun-Shiuan Chuang,  Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers

   In Findings of the Association for Computational Linguistics: NAACL 2024, 2024

   Abs PDF Code

   Accurately simulating human opinion dynamics is crucial for understanding a variety of societal phenomena, including polarization and the spread of misinformation. However, the agent-based models (ABMs) commonly used for such simulations often over-simplify human behavior. We propose a new approach to simulating opinion dynamics based on populations of Large Language Models (LLMs). Our findings reveal a strong inherent bias in LLM agents towards producing accurate information, leading simulated agents to consensus in line with scientific reality. This bias limits their utility for understanding resistance to consensus views on issues like climate change. After inducing confirmation bias through prompt engineering, however, we observed opinion fragmentation in line with existing agent-based modeling and opinion dynamics research. These insights highlight the promise and limitations of LLM agents in this domain and suggest a path forward: refining LLMs with real-world discourse to better simulate the evolution of human beliefs.

2. CogSci, ICLR’W

   

   The Wisdom of Partisan Crowds: Comparing Collective Intelligence in Humans and LLM-based Agents

   Yun-Shiuan Chuang, Nikunj Harlalka^*, Siddharth Suresh^*,  Agam Goyal, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers

   In Proceedings of the Annual Meeting of the Cognitive Science Society, 2024

   Abs PDF Code

   Human groups are able to converge on more accurate beliefs through deliberation, even in the presence of polarization and partisan bias – a phenomenon known as the "wisdom of partisan crowds." Generated agents powered by Large Language Models (LLMs) are increasingly used to simulate human collective behavior, yet few benchmarks exist for evaluating their dynamics against the behavior of human groups. In this paper, we examine the extent to which the wisdom of partisan crowds emerges in groups of LLM-based agents that are prompted to role-play as partisan personas (e.g., Democrat or Republican). We find that they not only display human-like partisan biases, but also converge to more accurate beliefs through deliberation as humans do. We then identify several factors that interfere with convergence, including the use of chain-of-thought prompt and lack of details in personas. Conversely, fine-tuning on human data appears to enhance convergence. These findings show the potential and limitations of LLM-based agents as a model of human collective intelligence.

3. CVPR

   

   Uncovering the Hidden Cost of Model Compression

   Diganta Misra^*, Muawiz Chaudhary^*,  Agam Goyal^*, Bharat Runwal^*, and Pin Yu Chen

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

   Abs PDF Supp Code

   In an age dominated by resource-intensive foundation models, the ability to efficiently adapt to downstream tasks is crucial. Visual Prompting (VP), drawing inspiration from the prompting techniques employed in Large Language Models (LLMs), has emerged as a pivotal method for transfer learning in the realm of computer vision. As the importance of efficiency continues to rise, research into model compression has become indispensable in alleviating the computational burdens associated with training and deploying over-parameterized neural networks. A primary objective in model compression is to develop sparse and/or quantized models capable of matching or even surpassing the performance of their over-parameterized, full-precision counterparts. Although previous studies have explored the effects of model compression on transfer learning, its impact on visual prompting-based transfer remains unclear. This study aims to bridge this gap, shedding light on the fact that model compression detrimentally impacts the performance of visual prompting-based transfer, particularly evident in scenarios with low data volume. Furthermore, our findings underscore the adverse influence of sparsity on the calibration of downstream visual-prompted models. However, intriguingly, we also illustrate that such negative effects on calibration are not present when models are compressed via quantization. This empirical investigation underscores the need for a nuanced understanding beyond mere accuracy in sparse and quantized settings, thereby paving the way for further exploration in Visual Prompting techniques tailored for sparse and quantized models.

4. EMNLP, NeurIPS’W

   

   Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks

   Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent V. Frigo, Sijia Yang, Dhavan V. Shah, Junjie Hu, and Timothy T. Rogers

   In Findings of the Association for Computational Linguistics: EMNLP 2024, Nov 2024

   Abs PDF

   Creating human-like large language model (LLM) agents is crucial for faithful social simulation. Having LLMs role-play based on demographic information sometimes improves human likeness but often does not. This study assessed whether LLM alignment with human behavior can be improved by integrating information from empirically-derived human belief networks. Using data from a human survey, we estimated a belief network encompassing 64 topics loading on nine non-overlapping latent factors. We then seeded LLM-based agents with an opinion on one topic, and assessed the alignment of its expressed opinions on remaining test topics with corresponding human data. Role-playing based on demographic information alone did not align LLM and human opinions, but seeding the agent with a single belief greatly improved alignment for topics related in the belief network, and not for topics outside the network. These results suggest a novel path for human-LLM belief alignment in work seeking to simulate and understand patterns of belief distributions in society.

5. arXiv

   

   Uncovering the Internet’s Hidden Values: An Empirical Study of Desirable Behavior Using Highly-Upvoted Content on Reddit

   Agam Goyal, Charlotte Lambert, and Eshwar Chandrasekharan

   arXiv preprint arXiv:2410.13036, Nov 2024

   Abs arXiv

   A major task for moderators of online spaces is norm-setting, essentially creating shared norms for user behavior in their communities. Platform design principles emphasize the importance of highlighting norm-adhering examples and explicitly stating community norms. However, norms and values vary between communities and go beyond content-level attributes, making it challenging for platforms and researchers to provide automated ways to identify desirable behavior to be highlighted. Current automated approaches of detecting desirability are limited to measures of prosocial behavior, but we do not know whether these measures fully capture the spectrum of what communities value. In this paper, we use upvotes, which express community approval, as a proxy for desirability and conduct an analysis of highly-upvoted comments across 85 popular sub-communities on Reddit. Using a large language model, we extract values from these comments and compile 97 macro, meso, and micro values based on their frequency across communities. Furthermore, we find that existing computational models for measuring prosociality were inadequate to capture 86 of the values we extracted. Finally, we show that our approach can not only extract most of the qualitatively-identified values from prior taxonomies, but also uncover new values that are actually encouraged in practice. This work has implications for improving moderator understanding of their community values, motivates the need for nuanced models of desirability beyond prosocial measures, and provides a framework that can supplement qualitative work with larger-scale content analyses.

6. arXiv

   

   SLM-Mod: Small Language Models Surpass LLMs at Content Moderation

   Xianyang Zhan^*,  Agam Goyal^*, Yilun Chen, Eshwar Chandrasekharan, and Koustuv Saha

   arXiv preprint arXiv:2410.13155, Nov 2024

   Abs arXiv

   Large language models (LLMs) have shown promise in many natural language understanding tasks, including content moderation. However, these models can be expensive to query in real-time and do not allow for a community-specific approach to content moderation. To address these challenges, we explore the use of open-source small language models (SLMs) for community-specific content moderation tasks. We fine-tune and evaluate SLMs (less than 15B parameters) by comparing their performance against much larger open- and closed-sourced models. Using 150K comments from 15 popular Reddit communities, we find that SLMs outperform LLMs at content moderation-11.5% higher accuracy and 25.7% higher recall on average across all communities. We further show the promise of cross-community content moderation, which has implications for new communities and the development of cross-platform moderation techniques. Finally, we outline directions for future work on language model based content moderation.

2023

1. CNA

   

   Tensor Decomposition to Capture Spatiotemporal Patterns of Coupled Oscillator and Opinion Dynamics

   Agam Goyal, and Hanbaek Lyu

   In Proceedings of the International Conference on Complex Networks and their Applications, Nov 2023

   Awarded PDF Code

   Oral Presentation

2. arXiv

   

   A latent linear model for nonlinear coupled oscillators on graphs

   Agam Goyal^*, Zhaoxing Wu^*, Richard P Yim, Binhao Chen, Zihong Xu, and Hanbaek Lyu

   arXiv preprint arXiv:2311.14910, Nov 2023

   Abs arXiv Code

   A system of coupled oscillators on an arbitrary graph is locally driven by the tendency to mutual synchronization between nearby oscillators, but can and often exhibit nonlinear behavior on the whole graph. Understanding such nonlinear behavior has been a key challenge in predicting whether all oscillators in such a system will eventually synchronize. In this paper, we demonstrate that, surprisingly, such nonlinear behavior of coupled oscillators can be effectively linearized in certain latent dynamic spaces. The key insight is that there is a small number of ‘latent dynamics filters’, each with a specific association with synchronizing and non-synchronizing dynamics on subgraphs so that any observed dynamics on subgraphs can be approximated by a suitable linear combination of such elementary dynamic patterns. Taking an ensemble of subgraph-level predictions provides an interpretable predictor for whether the system on the whole graph reaches global synchronization. We propose algorithms based on supervised matrix factorization to learn such latent dynamics filters. We demonstrate that our method performs competitively in synchronization prediction tasks against baselines and black-box classification algorithms, despite its simple and interpretable architecture.