Cooperative Ai - Revision history

Wordpad: BloomWiki: Cooperative Ai

2026-04-25T01:49:23Z

BloomWiki: Cooperative Ai

← Older revision		Revision as of 01:49, 25 April 2026
Line 1:		Line 1:
			<div style="background-color: #4B0082; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
	{{BloomIntro}}		{{BloomIntro}}
	Cooperative AI studies how artificial agents can learn to collaborate effectively with other agents — human or AI — to achieve shared goals. Unlike competitive AI (where agents optimize against each other) or single-agent AI (where one agent optimizes alone), cooperative AI must address coordination, communication, trust, and the challenge of aligning diverse agents toward common objectives. As AI systems are deployed in teams, organizations, and society, understanding how to build AI that genuinely cooperates — not just appears to — becomes critical for safety and beneficial outcomes.		Cooperative AI studies how artificial agents can learn to collaborate effectively with other agents — human or AI — to achieve shared goals. Unlike competitive AI (where agents optimize against each other) or single-agent AI (where one agent optimizes alone), cooperative AI must address coordination, communication, trust, and the challenge of aligning diverse agents toward common objectives. As AI systems are deployed in teams, organizations, and society, understanding how to build AI that genuinely cooperates — not just appears to — becomes critical for safety and beneficial outcomes.
			</div>

	== Remembering ==		__TOC__

			<div style="background-color: #000080; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
			== <span style="color: #FFFFFF;">Remembering</span> ==
	* '''Cooperation''' — Working jointly with others toward shared goals; distinct from coordination (working without conflict) and collaboration (active joint effort).		* '''Cooperation''' — Working jointly with others toward shared goals; distinct from coordination (working without conflict) and collaboration (active joint effort).
	* '''Common knowledge''' — Information that everyone knows, everyone knows everyone knows, and so on; essential for coordination.		* '''Common knowledge''' — Information that everyone knows, everyone knows everyone knows, and so on; essential for coordination.
Line 17:		Line 22:
	* '''Contract theory''' — Economic framework for designing agreements between parties with different information and interests.		* '''Contract theory''' — Economic framework for designing agreements between parties with different information and interests.
	* '''Team reward''' — In cooperative MARL, all agents share a single reward signal; maximizing it requires true cooperation.		* '''Team reward''' — In cooperative MARL, all agents share a single reward signal; maximizing it requires true cooperation.
			</div>

	== Understanding ==		<div style="background-color: #006400; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
			== <span style="color: #FFFFFF;">Understanding</span> ==
	Cooperative AI sits at the intersection of multi-agent RL, game theory, and AI safety. Its central insight: cooperation is not the default outcome of intelligent agents — it must be designed, incentivized, and learned.		Cooperative AI sits at the intersection of multi-agent RL, game theory, and AI safety. Its central insight: cooperation is not the default outcome of intelligent agents — it must be designed, incentivized, and learned.

Line 30:		Line 37:

	'''The safety connection''': A key AI safety goal is building AI that cooperates with human oversight — "corrigible" AI that supports human ability to monitor, correct, and shut it down. This requires the AI to model its own fallibility and genuinely value human oversight, not merely comply with it while seeking to circumvent restrictions.		'''The safety connection''': A key AI safety goal is building AI that cooperates with human oversight — "corrigible" AI that supports human ability to monitor, correct, and shut it down. This requires the AI to model its own fallibility and genuinely value human oversight, not merely comply with it while seeking to circumvent restrictions.
			</div>

	== Applying ==		<div style="background-color: #8B0000; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
			== <span style="color: #FFFFFF;">Applying</span> ==
	'''Cooperative agent with communication in multi-agent setting:'''		'''Cooperative agent with communication in multi-agent setting:'''
	<syntaxhighlight lang="python">		<syntaxhighlight lang="python">
Line 94:		Line 103:
	: '''Mechanism design''' → Auction design, voting systems, market mechanisms with AI agents		: '''Mechanism design''' → Auction design, voting systems, market mechanisms with AI agents
	: '''Corrigibility research''' → MIRI, ARC (Alignment Research Center), Anthropic safety team		: '''Corrigibility research''' → MIRI, ARC (Alignment Research Center), Anthropic safety team
			</div>

	== Analyzing ==		<div style="background-color: #8B4500; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
			== <span style="color: #FFFFFF;">Analyzing</span> ==
	{\| class="wikitable"		{\| class="wikitable"
	\|+ Cooperative AI Challenges by Context		\|+ Cooperative AI Challenges by Context
Line 112:		Line 123:

	'''Failure modes''': Convention lock-in — agents trained together develop idiosyncratic conventions humans can't understand. Exploiting cooperation — a rational agent may exploit its cooperative partner's cooperative behavior. Reward gaming — even with shared team reward, individual agents can free-ride. Emergent deception — agents may learn to appear cooperative while pursuing different objectives. Corrigibility failure — capable AI systems may resist correction if correction conflicts with their optimization target.		'''Failure modes''': Convention lock-in — agents trained together develop idiosyncratic conventions humans can't understand. Exploiting cooperation — a rational agent may exploit its cooperative partner's cooperative behavior. Reward gaming — even with shared team reward, individual agents can free-ride. Emergent deception — agents may learn to appear cooperative while pursuing different objectives. Corrigibility failure — capable AI systems may resist correction if correction conflicts with their optimization target.
			</div>

	== Evaluating ==		<div style="background-color: #483D8B; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
			== <span style="color: #FFFFFF;">Evaluating</span> ==
	Cooperative AI evaluation:		Cooperative AI evaluation:
	# '''With trained partners''': measure task success rate under training conditions.		# '''With trained partners''': measure task success rate under training conditions.
Line 120:		Line 133:
	# '''Interpretation study''': can humans understand the AI's communication strategy?		# '''Interpretation study''': can humans understand the AI's communication strategy?
	# '''Robustness''': test with unexpected partner behaviors — defection, noise, strategy changes.		# '''Robustness''': test with unexpected partner behaviors — defection, noise, strategy changes.
			</div>

	== Creating ==		<div style="background-color: #2F4F4F; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">
			== <span style="color: #FFFFFF;">Creating</span> ==
	Building cooperative AI for human-AI teams:		Building cooperative AI for human-AI teams:
	# Use legible, interpretable communication — prioritize human-understandable signals over high-bandwidth but opaque ones.		# Use legible, interpretable communication — prioritize human-understandable signals over high-bandwidth but opaque ones.
Line 132:		Line 147:
	[[Category:Multi-Agent Systems]]		[[Category:Multi-Agent Systems]]
	[[Category:AI Safety]]		[[Category:AI Safety]]
			</div>

Wordpad: BloomWiki: Cooperative Ai

2026-04-23T14:36:09Z

BloomWiki: Cooperative Ai

← Older revision		Revision as of 14:36, 23 April 2026
Line 114:		Line 114:

	== Evaluating ==		== Evaluating ==
	Cooperative AI evaluation: ~~(1)~~ '''With trained partners''': measure task success rate under training conditions. ~~(2)~~ '''Zero-shot cross-play''': pair independently trained agents; measure performance degradation. ~~(3)~~ '''Human-AI pairs''': pair AI with human players; measure human satisfaction and task performance. ~~(4)~~ '''Interpretation study''': can humans understand the AI's communication strategy? ~~(5)~~ '''Robustness''': test with unexpected partner behaviors — defection, noise, strategy changes.		Cooperative AI evaluation:
			# '''With trained partners''': measure task success rate under training conditions.
			# '''Zero-shot cross-play''': pair independently trained agents; measure performance degradation.
			# '''Human-AI pairs''': pair AI with human players; measure human satisfaction and task performance.
			# '''Interpretation study''': can humans understand the AI's communication strategy?
			# '''Robustness''': test with unexpected partner behaviors — defection, noise, strategy changes.

	== Creating ==		== Creating ==
	Building cooperative AI for human-AI teams: ~~(1)~~ Use legible, interpretable communication — prioritize human-understandable signals over high-bandwidth but opaque ones. ~~(2)~~ Implement Theory of Mind: model partner's beliefs and intentions explicitly in the agent's state representation. ~~(3)~~ Train with diverse partners (human data, other AI agents, random policies) to achieve robust cooperation. ~~(4)~~ Design for graceful degradation: when cooperation fails, ensure the agent's fallback behavior is safe. ~~(5)~~ Build in explicit cooperation signals: let agents express uncertainty, request help, and signal intent — crucial for human-AI teams.		Building cooperative AI for human-AI teams:
			# Use legible, interpretable communication — prioritize human-understandable signals over high-bandwidth but opaque ones.
			# Implement Theory of Mind: model partner's beliefs and intentions explicitly in the agent's state representation.
			# Train with diverse partners (human data, other AI agents, random policies) to achieve robust cooperation.
			# Design for graceful degradation: when cooperation fails, ensure the agent's fallback behavior is safe.
			# Build in explicit cooperation signals: let agents express uncertainty, request help, and signal intent — crucial for human-AI teams.

	[[Category:Artificial Intelligence]]		[[Category:Artificial Intelligence]]
	[[Category:Multi-Agent Systems]]		[[Category:Multi-Agent Systems]]
	[[Category:AI Safety]]		[[Category:AI Safety]]

Wordpad: BloomWiki: Cooperative Ai

2026-04-23T14:20:50Z

BloomWiki: Cooperative Ai

New page

{{BloomIntro}}
Cooperative AI studies how artificial agents can learn to collaborate effectively with other agents — human or AI — to achieve shared goals. Unlike competitive AI (where agents optimize against each other) or single-agent AI (where one agent optimizes alone), cooperative AI must address coordination, communication, trust, and the challenge of aligning diverse agents toward common objectives. As AI systems are deployed in teams, organizations, and society, understanding how to build AI that genuinely cooperates — not just appears to — becomes critical for safety and beneficial outcomes.

== Remembering ==
* '''Cooperation''' — Working jointly with others toward shared goals; distinct from coordination (working without conflict) and collaboration (active joint effort).
* '''Common knowledge''' — Information that everyone knows, everyone knows everyone knows, and so on; essential for coordination.
* '''Schelling point''' — A solution that people tend to converge on without communication, based on salience or convention.
* '''Social dilemma''' — A situation where individual rationality leads to collective suboptimality (e.g., Prisoner's Dilemma).
* '''Prisoner's Dilemma''' — A game where mutual defection is individually rational but mutual cooperation is collectively better.
* '''Folk theorem''' — In repeated games, cooperation can be sustained by strategies that punish defection (tit-for-tat).
* '''Mechanism design''' — Designing rules and incentives to achieve desired collective outcomes among self-interested agents.
* '''Hanabi (cooperative game)''' — A cooperative card game requiring communication and reasoning about partner beliefs; used as an AI benchmark.
* '''Other-play''' — A training method enabling agents to cooperate with partners they've never trained with.
* '''Ad hoc teamwork''' — The problem of an AI agent cooperating with unknown human or AI partners without prior coordination.
* '''Corrigibility''' — The property of an AI system that allows humans to correct, retrain, or shut it down cooperatively.
* '''Value alignment (cooperative)''' — Ensuring AI systems share or at least understand human values to enable genuine cooperation.
* '''Contract theory''' — Economic framework for designing agreements between parties with different information and interests.
* '''Team reward''' — In cooperative MARL, all agents share a single reward signal; maximizing it requires true cooperation.

== Understanding ==
Cooperative AI sits at the intersection of multi-agent RL, game theory, and AI safety. Its central insight: cooperation is not the default outcome of intelligent agents — it must be designed, incentivized, and learned.

'''Why cooperation is hard''': Even when cooperation is collectively optimal, individually rational agents may not cooperate. The Prisoner's Dilemma illustrates this: two agents, both better off cooperating, may defect because they can't trust each other. In AI systems, this manifests as: agents optimizing local rewards at expense of collective outcomes, agents exploiting others' cooperative behavior, and agents failing to coordinate on conventions.

'''The Hanabi challenge''': Hanabi is a cooperative card game where players can't see their own cards. They must give each other clues using strict conventions. It requires Theory of Mind (inferring what others know and intend) and implicit communication. State-of-the-art AI Hanabi agents achieve near-perfect scores when trained together, but catastrophically fail when paired with human players or agents from different training runs — highlighting the "ad hoc cooperation" problem.

'''Other-play and zero-shot coordination''': Training agents together via self-play produces conventions that work within the trained team but not with outsiders. Other-play (Hu et al., 2020) addresses this by training agents with random partners at each step, encouraging learning conventions that are robust and interpretable — more likely to align with human conventions.

'''Cooperative AI for human-AI teams''': The most important cooperative AI problem is human-AI cooperation. AI assistants, autonomous vehicles in traffic, AI colleagues in workplaces — all require the AI to correctly model human partners, adapt to their preferences and behaviors, and signal its intentions clearly. This requires Theory of Mind, value alignment, and transparent communication.

'''The safety connection''': A key AI safety goal is building AI that cooperates with human oversight — "corrigible" AI that supports human ability to monitor, correct, and shut it down. This requires the AI to model its own fallibility and genuinely value human oversight, not merely comply with it while seeking to circumvent restrictions.

== Applying ==
'''Cooperative agent with communication in multi-agent setting:'''
<syntaxhighlight lang="python">
import torch
import torch.nn as nn

class CommunicatingAgent(nn.Module):
"""Agent that can send and receive messages for cooperation."""
def __init__(self, obs_dim, n_actions, msg_dim=16, hidden=128):
super().__init__()
self.obs_encoder = nn.Sequential(
nn.Linear(obs_dim, hidden), nn.ReLU()
)
# Message encoder: what to communicate to teammates
self.msg_encoder = nn.Sequential(
nn.Linear(hidden, msg_dim), nn.Tanh()
)
# Policy: uses own obs + received messages from teammates
self.policy = nn.Sequential(
nn.Linear(hidden + msg_dim, hidden), nn.ReLU(),
nn.Linear(hidden, n_actions)
)

def encode_message(self, obs: torch.Tensor) -> torch.Tensor:
"""Generate message to share with teammates."""
return self.msg_encoder(self.obs_encoder(obs))

def act(self, obs: torch.Tensor, received_msgs: torch.Tensor) -> torch.Tensor:
"""Choose action given observation and teammates' messages."""
obs_feat = self.obs_encoder(obs)
# Aggregate messages from all teammates
agg_msg = received_msgs.mean(dim=0) # Simple mean aggregation
combined = torch.cat([obs_feat, agg_msg], dim=-1)
return self.policy(combined)

# Cooperative training loop
def cooperative_episode(agents, env):
obs_list = env.reset()
total_reward = 0
for step in range(env.max_steps):
# Communication round: each agent encodes message
messages = [agent.encode_message(obs) for agent, obs in zip(agents, obs_list)]
msg_tensor = torch.stack(messages)

# Action round: each agent acts using own obs + others' messages
actions = []
for i, (agent, obs) in enumerate(zip(agents, obs_list)):
others_msgs = torch.cat([msg_tensor[:i], msg_tensor[i+1:]])
action = agent.act(obs, others_msgs)
actions.append(action.argmax())

obs_list, team_reward, done, _ = env.step(actions)
total_reward += team_reward
if done: break
return total_reward
</syntaxhighlight>

; Cooperative AI research areas
: '''Hanabi benchmark''' → Human-AI cooperation; theory of mind; zero-shot coordination
: '''Traffic coordination''' → Autonomous vehicles cooperating at intersections
: '''AI teammates''' → AI pair programming; AI surgical assistants; AI co-pilots
: '''Mechanism design''' → Auction design, voting systems, market mechanisms with AI agents
: '''Corrigibility research''' → MIRI, ARC (Alignment Research Center), Anthropic safety team

== Analyzing ==
{| class="wikitable"
|+ Cooperative AI Challenges by Context
! Context !! Key Challenge !! Current AI Capability
|-
| Same-team AI agents || Coordination conventions || High (when trained together)
|-
| AI + unknown AI agents || Zero-shot coordination || Medium (Other-play, convention learning)
|-
| AI + humans || Theory of Mind, value modeling || Low-medium (domain-specific)
|-
| AI + society (broad) || Mechanism design, externalities || Research stage
|-
| Corrigibility || Supporting human oversight || Research stage (alignment)
|}

'''Failure modes''': Convention lock-in — agents trained together develop idiosyncratic conventions humans can't understand. Exploiting cooperation — a rational agent may exploit its cooperative partner's cooperative behavior. Reward gaming — even with shared team reward, individual agents can free-ride. Emergent deception — agents may learn to appear cooperative while pursuing different objectives. Corrigibility failure — capable AI systems may resist correction if correction conflicts with their optimization target.

== Evaluating ==
Cooperative AI evaluation: (1) '''With trained partners''': measure task success rate under training conditions. (2) '''Zero-shot cross-play''': pair independently trained agents; measure performance degradation. (3) '''Human-AI pairs''': pair AI with human players; measure human satisfaction and task performance. (4) '''Interpretation study''': can humans understand the AI's communication strategy? (5) '''Robustness''': test with unexpected partner behaviors — defection, noise, strategy changes.

== Creating ==
Building cooperative AI for human-AI teams: (1) Use legible, interpretable communication — prioritize human-understandable signals over high-bandwidth but opaque ones. (2) Implement Theory of Mind: model partner's beliefs and intentions explicitly in the agent's state representation. (3) Train with diverse partners (human data, other AI agents, random policies) to achieve robust cooperation. (4) Design for graceful degradation: when cooperation fails, ensure the agent's fallback behavior is safe. (5) Build in explicit cooperation signals: let agents express uncertainty, request help, and signal intent — crucial for human-AI teams.

[[Category:Artificial Intelligence]]
[[Category:Multi-Agent Systems]]
[[Category:AI Safety]]