Think-In-Games (TiG): Bridging Declarative and Procedural Knowledge in LLMs via Reinforcement Learning on Honor of Kings

Forum topic · ✨步子哥 · 2025-10-03

Summary

Tencent's Think-In-Games (TiG) framework enables large language models to act and explain decisions in the MOBA game Honor of Kings by reframing reinforcement learning as a language modeling task. TiG uses a relabeling algorithm that performs backward filling and priority override on real match data to assign macro-action labels (e.g., Push Mid Lane, Secure Dragon, Defend Base) to JSON-encoded game states, yielding dense and consistent training signals. Training combines supervised fine-tuning on DeepSeek-R1 reasoning traces with online Group Relative Policy Optimization (GRPO), which maximizes advantages of generated samples while constraining KL divergence from a reference model, using a binary reward based on action match. A 14B-parameter Qwen-3-14B model reached 90.91% macro-action accuracy, outperforming the 671B-parameter DeepSeek-R1 (86.67%). The work demonstrates lower data and compute requirements, transparent chain-of-thought reasoning, and potential transfer to robotics and virtual assistants.

Overview

Tencent researchers introduced Think-In-Games (TiG), a framework that lets large language models (LLMs) both act in and explain decisions made within the MOBA game Honor of Kings (王者荣耀). TiG reframes reinforcement learning (RL) decision-making as a language modeling task, bridging declarative knowledge ("knowing why") and procedural knowledge ("knowing how").

The Knowledge Gap in LLMs

LLMs excel at declarative knowledge — mathematical theorems, code, history, literature — but typically lack procedural knowledge needed for interactive tasks such as navigation, causal reasoning, or gameplay. Children acquire procedural skills through play; LLMs are trained on static text and rarely receive real-time environmental feedback. Traditional game AI (search, heuristics, RL) achieves strong results but is opaque and resource-hungry. TiG combines the world knowledge and reasoning of LLMs with the adaptive feedback loop of RL.

Data Collection: Relabeling Algorithm

High-quality data is the foundation. The team sampled real Honor of Kings matches, but raw game states are continuous while macro-actions are episodic. A relabeling algorithm solves this:

1. Backward filling — propagate a known macro-action label backward across a window of preceding frames. 2. Priority override — when multiple actions could apply, choose the one with highest priority (e.g., emergency defense outranks routine advance).

The macro-action space contains 40 predefined team-level objectives, including Push Mid Lane, Secure Dragon, Defend Base, and Team Fight. Game states are encoded in JSON containing visible information: hero attributes, lane state, tower status, resources, and vision — but excluding hidden enemy data, preserving imperfect-information realism.

Example reasoning chain from the paper: the model observes that mid-lane tower protection has expired, identifies the low-HP mid tower as a priority target, plans joint push with ally Jiang Ziya, warns of possible enemy ambush, and issues the command: *"Push the enemy mid-lane first tower with Jiang Ziya; watch for enemy ambushes."*

GRPO: The Optimization Engine

TiG's core is Group Relative Policy Optimization (GRPO), an online RL method for LLMs:

$$L(\theta) = \mathbb{E}_{x \sim D}\left[ \log \sigma\left( \frac{1}{K} \sum_{k=1}^{K} \big( r(x, y_k) - r(x, y_{ref}) \big) \right) \right] + \beta \cdot D_{KL}(p_\theta \| p_{ref})$$

$ \theta $: model parameters
$ D $: dataset
$ r $: reward function
$ y_k $: generated samples; $ y_{ref} $: reference outputs
$ \sigma $: sigmoid; $ \beta $: KL penalty coefficient

The first term encourages high-reward samples; the second prevents drift from the reference policy. The reward is a simple binary signal (1 if the predicted action matches human gameplay, else 0), avoiding complex reward engineering.

Training proceeds in two stages: 1. Supervised fine-tuning (SFT) on reasoning traces distilled from DeepSeek-R1. 2. Online RL with GRPO on real game data for 2000 iterations.

Experimental Results

Models tested: Qwen-2.5-7B-Instruct, Qwen-2.5-14B-Instruct, Qwen-2.5-32B-Instruct, Qwen-3-14B-Instruct, and DeepSeek-R1 (671B). Metric: macro-action accuracy vs. human labels.

| Model | No training | SFT | GRPO | SFT+GRPO | |-------|-------------|------|-------|-----------| | Qwen-2.5-7B | 45.12% | 58.76% | 62.34% | 75.89% | | Qwen-2.5-14B | 53.25% | 68.90% | 72.45% | 83.12% | | Qwen-2.5-32B | 66.67% | 78.23% | 80.56% | 86.84% | | Qwen-3-14B | 70.00% | 82.50% | 85.71% | 90.91% | | DeepSeek-R1 | — | — | — | 86.67% |

Qwen-3-14B with SFT+GRPO reached 90.91% accuracy, exceeding the 671B-parameter DeepSeek-R1 baseline (86.67%) at a fraction of the compute.

Implications and Limitations

Advantages: transparent reasoning chains, low data/compute cost, interpretable decisions.
Current limits: restricted to macro-actions; micro-actions (precise skill shots) are out of scope; dependence on human-labeled data.
Future directions: extending to micro-level control, self-supervised label discovery, and transfer to robotics and virtual assistants.

References

1. Yi Liao et al. *Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models.* arXiv:2508.21365 (2025). 2. Rohan Paul. *X post.* https://x.com/rohanpaul_ai/status/1962499431137493195 3. Tencent AI Lab. *Honor of Kings: Macro-Strategy Dataset and Benchmarks.* Internal report, 2025. 4. OpenAI. *Reinforcement Learning from Human Feedback in Interactive Environments.* NeurIPS, 2024. 5. DeepSeek Team. *DeepSeek-R1 Technical Report.* 2025.

Reference image from the X thread: !TiG framework applied in Honor of Kings