DNQ: Deep Nash Q-Network for Partially Observable n-Player Games

Paper Overview

Research area: Machine Learning Authors: Qintong Xie, Edward Koh, Xavier Cadet Published: 2026-06-04 arXiv: 2606.06480

Summary

Many real-world competitive systems require multiple decision-makers to act simultaneously under shared constraints, limited information, and repeated interaction, as in auctions, resource allocation, and security competition. The authors study multi-turn simultaneous bidding as a controlled testbed for such problems and propose DNQ, a solver-in-the-loop equilibrium supervision framework for training bidding agents.

How DNQ Works

DNQ alternates between four stages:

1. Trajectory collection 2. Critic-based payoff estimation 3. Equilibrium computation 4. Policy imitation

At each visited state, a shared critic predicts either pairwise payoff matrices or an exact N-player payoff tensor. An external solver computes equilibrium strategies, and agents are trained by minimizing the KL divergence between their masked policies and the solver-derived equilibrium targets.

Scalability: Pairwise vs. Exact

The paper focuses on a scalable pairwise formulation, which compared with the exact formulation:

• Substantially reduces equilibrium-solving cost
• Reduces training time
• Amortizes payoff learning across agents and states via the shared critic

Experiments compare pairwise and exact variants. The pairwise method scales to more agents, while the exact method becomes computationally infeasible as the joint game grows.

Takeaway

These results demonstrate the trade-off between policy fidelity and scalability in repeated competitive environments.

Original Abstract (excerpt)

> Many real-world competitive systems require multiple decision-makers to act simultaneously under shared constraints, limited information, and repeated interaction, as in auctions, resource allocation, and security competition. We study multi-turn simultaneous bidding as a controlled testbed for such problems and propose DNQ, a solver-in-the-loop equilibrium supervision framework for training bidding agents. DNQ alternates between trajectory collection, critic-based payoff estimation, equilibrium computation, and policy imitation. At each visited state, a shared critic predicts either pairwise payoff matrices or an exact N-player payoff tensor, an external solver computes equilibrium strategies, and the agents are trained by minimizing the KL divergence between their masked policies and the...

--- *Auto-collected on 2026-06-07*