Summary
This paper studies multi-teacher knowledge distillation for low-resource abstractive summarization from a reliability-aware perspective. The authors introduce EWAD (Entropy Weighted Agreement Aware Distillation), a token-level mechanism that routes supervision between teacher distillation and gold supervision based on inter-teacher agreement, and CPDP (Capacity Proportional Divergence Preservation), a geometric constraint on the student's position relative to heterogeneous teachers. Experiments across two Bangla datasets, 13 BanglaT5 ablations, and eight Qwen2.5 experiments show that logit-level KD provides the most reliable gains, while more complex distillation improves semantic similarity for short summaries but degrades longer outputs. Cross-lingual pseudo-label KD across ten languages retains 71-122 percent of teacher performance. The paper is available on arXiv (2604.03192).
论文概要
研究领域: NLP
作者: Dipto Sumit, Ankan Kumar Roy, Sadia Khair Rodela 等
发布时间: 2026-04-03
arXiv: 2604.03192
English Translation
We study multiteacher knowledge distillation for low-resource abstractive summarization from a reliability-aware perspective. We introduce EWAD (Entropy Weighted Agreement Aware Distillation), a token-level mechanism that routes supervision between teacher distillation and gold supervision based on inter-teacher agreement, and CPDP (Capacity Proportional Divergence Preservation), a geometric constraint on the student position relative to heterogeneous teachers.
Across two Bangla datasets, 13 BanglaT5 ablations, and eight Qwen2.5 experiments, we find that logit-level KD provides the most reliable gains, while more complex distillation improves semantic similarity for short summaries but degrades longer outputs. Cross-lingual pseudo-label KD across ten languages retains 71-122 percent of teacher performance.
---
*Auto-collected on 2026-04-06*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177169587