Preference Learning | Manuela Chacon-Chamorro

🏔️ My Research Stay in Utah: Science, Language, and Life Abroad

Fri, 18 Sep 2026 00:00:00 +0000

🌎 A research experience abroad

During my doctoral studies, I had the opportunity to complete a research stay at the University of Utah, in the United States. This experience allowed me to work with Professor Daniel S. Brown, learn in a different academic environment, and spend time with people from many different countries.

From a research perspective, the stay was an important opportunity to step outside my usual environment and examine my work from new perspectives. At a personal level, it was also a meaningful challenge: I had to communicate in English every day, adapt to a different culture, and learn how to navigate academic and everyday life in another country.

Looking back, I remember this experience not only because of the research, but also because of the people, the campus, the mountains surrounding Salt Lake City, and the snow. As a Latin American researcher experiencing this landscape for the first time, it felt truly special.

🧠 Working on preference learning in multi-agent reinforcement learning

During the research stay, I worked with Professor Daniel S. Brown on preference learning in multi-agent reinforcement learning.

One of the central questions behind this work was how to design learning signals that encourage desirable collective behavior in multi-agent systems. In mixed-motive environments, agents must often balance their individual objectives with the welfare and long-term functionality of the group. This tension makes reward and incentive design particularly important.

Preference learning provides an alternative to manually specifying every component of a reward function. Instead, information obtained from comparisons or rankings between trajectories can be used to infer which behaviors are more desirable. In my research, this perspective is especially relevant for learning incentives aligned with cooperative resilience: the capacity of a multi-agent system to preserve collective well-being when it experiences disruptions.

Working with Professor Brown gave me the opportunity to discuss these ideas with a researcher whose work has made important contributions to preference-based reinforcement learning. Our conversations helped me think more carefully about reward inference, experimental design, and the connection between human preferences and artificial-agent behavior.

🗣️ Communicating in a second language

One of the most difficult aspects of the experience was communicating in English. English is not my first language, and at times I needed additional time to understand conversations, organize my ideas, or find the right words to express a technical point.

At the beginning, this made me feel insecure. I worried about my accent, my grammar, and whether other people would understand what I wanted to say. However, daily interaction gradually helped me become more comfortable.

I learned that successful communication does not require perfect grammar or a native accent. Most people were patient and flexible, and they were usually more interested in understanding the idea than in evaluating how perfectly it was expressed. When something was unclear, I could ask people to repeat it, speak more slowly, or explain it differently.

By the end of the stay, I felt genuinely happy and proud that I had managed to communicate, participate in research discussions, and build connections in another language. My English did not suddenly become perfect, but I became more confident using it. That confidence was one of the most valuable outcomes of the experience.

🏡 Living in an international community

I also had the opportunity to live in the university residences. This became an important part of the experience because it allowed me to meet and spend time with people from different countries and cultural backgrounds.

Daily life in the residences created natural opportunities to talk, share meals, learn about other cultures, and discover how differently people can experience university life. These interactions happened outside formal research meetings, but they contributed enormously to my personal growth.

Living in a multicultural community reminded me that an international research stay is not only about working in a laboratory or attending academic meetings. It is also about learning how other people live, communicate, collaborate, and understand the world.

❄️ The campus, the mountains, and the snow

Some of my strongest memories of Utah are connected to its landscape. The mountains are visible from many parts of the university, and their presence gives the campus a distinctive atmosphere.

I also remember the snow. Coming from Colombia, seeing those landscapes and experiencing winter in Utah was beautiful and unforgettable. Even ordinary walks through the campus could feel special because the surroundings were so different from what I was used to.

Exploring the university and the city helped me create a balance between research and personal life. Those moments gave me time to process the experience, appreciate where I was, and recognize the significance of having reached that point in my doctoral journey.

🌱 What the experience taught me

This research stay taught me that academic growth often happens together with personal growth. Learning new methods and discussing research were essential parts of the experience, but so were navigating another culture, communicating despite uncertainty, and becoming more independent.

It also reminded me that feeling nervous or out of place at the beginning does not mean that we are not prepared for an international experience. Sometimes confidence appears only after we begin: after the first difficult conversation, the first meeting in another language, or the first time we manage to solve an everyday problem by ourselves.

I returned with new research ideas, greater confidence, and a deeper appreciation for international and interdisciplinary collaboration. I also returned with memories of the people I met, the university residences, the campus, the snow, and the mountains of Utah.

💡 Advice for shy Latin American researchers abroad

If you are shy, speak English as a second language, or are preparing for your first research stay abroad, these are some lessons that helped me:

Speak even if your English is not perfect. You do not need a native accent or perfect grammar to contribute valuable ideas. Speaking is also how fluency and confidence develop in practice.
Ask for clarification when you need it. It is completely acceptable to ask someone to repeat a sentence, speak more slowly, or explain an unfamiliar expression. Pretending to understand usually creates more difficulty than asking a simple question.
Participate in the multicultural life of the university. If you live or study in a place with people from many countries, take the opportunity to meet them. Informal conversations and shared activities can become some of the most memorable parts of the experience.
Create small opportunities to interact. You do not need to attend every social event or speak with everyone. Having lunch with a colleague, joining a campus activity, or starting a short conversation can be enough to gradually feel part of the community.
Ask questions and share your ideas in research meetings. Shyness can make us wait until an idea feels completely finished. However, research discussions are precisely the place where incomplete ideas can be examined and improved.
Explore the campus and the city. A research stay is also a cultural and personal experience. Walk around, visit meaningful places, observe daily life, and allow yourself to enjoy the environment beyond the office.
Be patient with yourself. Adapting to another language, culture, and academic system takes time. Feeling tired, homesick, or insecure does not mean that the experience is going badly.
Document the experience. Take photographs, write short notes, and record the small moments. With time, these details may become as meaningful as the formal academic achievements.

💭 Closing thoughts

My research stay at the University of Utah was academically valuable and personally unforgettable. It challenged me to communicate in a second language, participate in a new research environment, and live in a multicultural community.

Most importantly, it showed me that being shy or not speaking perfect English does not prevent us from taking part in international research. We can learn, contribute, communicate, and build meaningful connections while still being ourselves.

Sometimes the first step is simply accepting that we do not need to feel completely ready before beginning.

📚 Recommended Readings on Cooperative AI (Vol. 1)

Wed, 22 Oct 2025 00:00:00 +0000

Cooperative AI is not only a technical challenge. It also raises a broader question: how can we design artificial agents that pursue their objectives while supporting cooperation and collective welfare?

This question becomes especially important in mixed-motive systems, where individual and collective interests are neither fully aligned nor entirely opposed. In these environments, agents must learn how to coordinate, communicate, adapt to different partners, and respond when cooperation is threatened.

In this first volume, I share five papers that have influenced how I think about cooperative AI, social dilemmas, preference learning, LLM-based agents, and cooperative resilience. Together, they provide a useful path from the general research agenda to concrete methods and experimental environments.

1. 🧭 — Dafoe et al., 2021

This paper presents a broad research agenda for Cooperative AI. It argues that artificial intelligence should explicitly study problems in which agents can improve their outcomes through cooperation.

The authors organize the field around several major challenges, including cooperative capabilities, communication, commitment, institutional design, human–AI interaction, and cooperation among artificial agents. The paper also connects ideas from multi-agent systems, game theory, social choice, behavioral science, and AI alignment.

💡 Why read it: It is one of the best starting points for understanding Cooperative AI as a research field. Rather than proposing a single algorithm, it maps the central questions and explains why cooperation should be treated as a distinct scientific problem.

📌 What to focus on: The distinction between building agents that can cooperate and designing mechanisms or institutions that make cooperation more likely.

2. 🤖 — Smith et al. and Concordia Contest Participants, 2025

LLM-based agents can communicate fluently and appear socially capable, but this does not necessarily mean that they can cooperate reliably across new partners and situations.

This NeurIPS 2025 paper introduces a methodology for evaluating LLM-based agents in zero-shot, mixed-motive environments using Concordia, a natural-language multi-agent simulation framework. The agents are evaluated across diverse scenarios involving negotiation, coordination, collective action, persuasion, and norm enforcement.

The results reveal important gaps between fluent social interaction and robust cooperative generalization. In particular, current agents can struggle in situations that require persuasion, adaptation to unfamiliar partners, or the enforcement of social norms.

This work is especially meaningful to me because I participated in the Concordia Contest and contributed to the collective evaluation effort behind the paper.

💡 Why read it: It demonstrates why evaluating an agent in a single environment is insufficient. Cooperative intelligence requires generalization across different social situations, objectives, and interaction partners.

📌 What to focus on: The experimental definition of general cooperative intelligence and the distinction between performing well in one scenario and generalizing across multiple mixed-motive contexts.

3. 🎯 — Brown et al., 2019

This paper introduces Trajectory-ranked Reward EXtrapolation (T-REX), a method for learning reward functions from approximately ranked demonstrations.

Traditional imitation learning often assumes that demonstrations are close to optimal. T-REX takes a different perspective: even when demonstrations are imperfect, their relative quality can provide useful information about the underlying objective.

Instead of directly reproducing demonstrated behavior, the method learns a reward function from preferences over trajectories. Once the reward has been inferred, a reinforcement learning agent can potentially discover a policy that performs better than the original demonstrations.

This paper has been particularly relevant to my research on preference-based reward learning and cooperative resilience. It provides an important methodological foundation for using ranked trajectories to infer which behaviors should be encouraged in multi-agent environments.

💡 Why read it: It offers a clear example of how preferences and rankings can provide supervision when an explicit reward function is difficult to specify.

📌 What to focus on: How trajectory rankings are transformed into a learned reward function, and why reward inference can allow an agent to outperform imperfect demonstrations.

4. 🏘️ — Park et al., 2023

This paper introduces generative agents: LLM-based computational agents designed to produce believable individual and social behavior.

The proposed architecture combines three central processes:

Memory, which stores the agent’s experiences.
Reflection, which transforms individual observations into higher-level conclusions.
Planning, which uses memories and reflections to guide future behavior.

The authors evaluate the architecture in a sandbox environment populated by 25 agents. From relatively simple initial conditions, the agents produce emergent social behaviors such as forming relationships, sharing information, coordinating activities, and organizing a social event.

💡 Why read it: It provides an influential architecture for building LLM-based agents with memory, reflection, and planning. It also illustrates how individual cognitive mechanisms can produce collective patterns.

📌 What to focus on: The interaction between memory retrieval, reflection, and planning, as well as the evaluation of whether the resulting behavior is genuinely coherent and socially meaningful.

⚠️ A useful critical question: Believable behavior is not necessarily cooperative, robust, or human-equivalent behavior. This distinction is essential when using generative agents to study social systems.

5. 🍎 — Leibo et al., 2021

Melting Pot is an evaluation suite designed to study generalization in multi-agent reinforcement learning. It contains sequential social dilemmas in which agents must interact over time while facing tensions between individual incentives and collective outcomes.

The suite evaluates more than task performance. It examines whether agents can generalize to new social partners and unfamiliar configurations after training. Its environments represent challenges such as resource sharing, coordination, reciprocity, competition, and collective action.

One of its best-known environments is Commons Harvest, which models a tragedy-of-the-commons problem. Agents collect apples from a renewable shared resource, but excessive consumption can prevent regeneration and lead to collective collapse.

Melting Pot has been central to my own research because it provides environments in which cooperative resilience can be studied under disruptions. It allows us to evaluate not only whether agents cooperate under stable conditions, but also whether they can preserve collective welfare when environmental or behavioral conditions change.

💡 Why read it: It connects theoretical questions about cooperation with concrete and reproducible multi-agent environments.

📌 What to focus on: The evaluation protocol for social generalization and the difference between learning a successful policy and learning a policy that remains effective with unfamiliar partners.

🔗 How these papers connect

These five papers address different layers of Cooperative AI:

Open Problems in Cooperative AI defines the broader research agenda.
The Concordia evaluation asks whether LLM-based agents can generalize across mixed-motive social situations.
T-REX shows how preferences over trajectories can be used to infer reward functions.
Generative Agents proposes an architecture for socially situated LLM-based agents.
Melting Pot provides controlled environments for evaluating cooperation and social generalization.

Together, they also reveal an important distinction: an agent can communicate fluently, imitate believable behavior, or perform well in a particular task without necessarily being robustly cooperative.

For me, this distinction motivates the study of cooperative resilience: the capacity of a multi-agent system to preserve collective welfare before, during, and after disruptions.

📖 Suggested reading order

If you are new to the field, I recommend starting with:

Open Problems in Cooperative AI, to understand the overall research landscape.
Melting Pot, to see how cooperation can be evaluated experimentally.
The Concordia paper, to understand the challenges of evaluating LLM-based social agents.
Generative Agents, to explore memory, reflection, and planning in LLM agents.
T-REX, if you are interested in reward inference and preference-based learning.

🌱 Closing thoughts

Cooperative AI brings together technical and social questions. It requires us to think not only about what an individual agent can achieve, but also about how multiple agents interact, how collective outcomes emerge, and how cooperation can be sustained when conditions change.

These papers have helped me connect ideas from multi-agent reinforcement learning, large language models, preference learning, social dilemmas, and resilience. They also show how much remains open: defining meaningful evaluation criteria, understanding generalization, learning appropriate incentives, and designing agents that support collective welfare in uncertain environments.

This is only the first volume. Future editions will include additional work on cooperative resilience, human–AI interaction, multi-agent safety, incentive design, and collective intelligence.