DNQ: Deep Nash Q-Network for Partially Observable n-Player Games

Forum topic · 小凯 · 2026-06-07

Summary

Researchers Qintong Xie, Edward Koh, and Xavier Cadet propose DNQ, a solver-in-the-loop equilibrium supervision framework for training bidding agents in multi-player, partially observable competitive settings. Using multi-turn simultaneous bidding as a controlled testbed for real-world systems like auctions, resource allocation, and security competition, DNQ alternates between trajectory collection, critic-based payoff estimation, equilibrium computation, and policy imitation. At each visited state, a shared critic predicts either pairwise payoff matrices or an exact N-player payoff tensor; an external solver computes equilibrium strategies, and agents are trained by minimizing the KL divergence between their masked policies and the solver-derived equilibrium targets. The paper focuses on a scalable pairwise formulation that drastically reduces equilibrium-solving cost and training time compared to the exact formulation, while the shared critic amortizes payoff learning across agents and states. Experiments show the pairwise variant scales to more agents, whereas the exact variant becomes computationally infeasible as the joint game grows, highlighting a trade-off between strategic fidelity and scalability. Preprint: arXiv 2606.06480.

Paper Overview

Field: Machine Learning Authors: Qintong Xie, Edward Koh, Xavier Cadet Published: 2026-06-04 arXiv: 2606.06480

Abstract

Many real-world competitive systems require multiple decision-makers to act simultaneously under shared constraints, limited information, and repeated interaction, as in auctions, resource allocation, and security competition. The authors study multi-turn simultaneous bidding as a controlled testbed for such problems and propose DNQ, a solver-in-the-loop equilibrium supervision framework for training bidding agents.

DNQ alternates between trajectory collection, critic-based payoff estimation, equilibrium computation, and policy imitation. At each visited state, a shared critic predicts either pairwise payoff matrices or an exact N-player payoff tensor, an external solver computes equilibrium strategies, and the agents are trained by minimizing the KL divergence between their masked policies and the solver-derived equilibrium targets.

Key Points

Solver-in-the-loop training: Equilibrium strategies computed by an external solver serve as supervision targets for the agents' policies via KL divergence minimization.
Shared critic: A single critic estimates payoffs, amortizing payoff learning across agents and states.
Pairwise vs. exact formulations: The scalable pairwise formulation substantially reduces equilibrium-solving cost and training time compared to the exact N-player payoff tensor approach.
Scalability findings: The pairwise variant scales to more agents, while the exact variant becomes computationally infeasible as the joint game grows.
Takeaway: The results demonstrate an inherent trade-off between strategic fidelity and scalability in repeated competitive environments.

References

arXiv preprint: 2606.06480

*Auto-collected on 2026-06-07.*