ELLA Efficient Lifelong Learning for LLMs

✨步子哥 · 2026-01-09T11:42:02+00:00

ELLA Framework Poster @import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600;800&family=JetBrains+Mono:wght@400;700&display=swap'); * { box-sizing: border-box; margin: 0; padding: 0; } body { width: 720px; height: 960px; background: #050511; font-family: 'Inter', sans-serif; color: #e2e8f0; overflow: hidden; display: flex; flex-direction: column; position: relative; } /* Background Effects */ .bg-gradient { position: absolute; top: 0; left: 0; width: 100%; height: 100%; background: radial-gradient(circle at 10% 20%, rgba(76, 29, 149, 0.25) 0%, transparent 40%), radial-gradient(circle at 90% 80%, rgba(6, 182, 212, 0.15) 0%, transparent 40%); z-index: 0; } .grid-lines { position: absolute; top: 0; left: 0; width: 100%; height: 100%; background-image: linear-gradient(rgba(255, 255, 255, 0.03) 1px, transparent 1px), linear-gradient(90deg, rgba(255, 255, 255, 0.03) 1px, transparent 1px); background-size: 40px 40px; z-index: 1; } /* Content Container */ .container { position: relative; z-index: 10; padding: 40px; height: 100%; display: flex; flex-direction: column; gap: 24px; } /* Header */ header { display: flex; justify-content: space-between; align-items: flex-start; border-bottom: 1px solid rgba(255, 255, 255, 0.1); padding-bottom: 20px; } .title-block h1 { font-size: 64px; font-weight: 800; line-height: 1; background: linear-gradient(135deg, #fff 0%, #94a3b8 100%); -webkit-background-clip: text; -webkit-text-fill-color: transparent; letter-spacing: -2px; margin-bottom: 8px; } .title-block p { font-size: 18px; color: #64748b; font-weight: 400; text-transform: uppercase; letter-spacing: 2px; } .badge { background: rgba(6, 182, 212, 0.1); border: 1px solid rgba(6, 182, 212, 0.3); color: #22d3ee; padding: 6px 12px; border-radius: 20px; font-size: 12px; font-weight: 600; text-transform: uppercase; } /* Main Grid */ .grid-layout { display: grid; grid-template-columns: 1fr 1fr; grid-template-rows: auto auto 1fr; gap: 20px; flex-grow: 1; } .full-width { grid-column: 1 / -1; } /* Cards */ .card { background: rgba(255, 255, 255, 0.03); border: 1px solid rgba(255, 255, 255, 0.08); border-radius: 16px; padding: 20px; backdrop-filter: blur(10px); display: flex; flex-direction: column; gap: 12px; transition: transform 0.3s ease; } .card-title { font-size: 16px; font-weight: 600; color: #a5b4fc; display: flex; align-items: center; gap: 8px; } .card-title::before { content: ''; display: block; width: 4px; height: 16px; background: #6366f1; border-radius: 2px; } .card-content { font-size: 14px; line-height: 1.6; color: #cbd5e1; } /* Highlighted Text */ .highlight { color: #fff; font-weight: 600; } /* Math Section */ .math-box { background: #0f172a; border-radius: 8px; padding: 15px; font-family: 'JetBrains Mono', monospace; color: #4ade80; font-size: 13px; text-align: center; border: 1px dashed #334155; margin-top: 5px; } /* Visual Representation of Subspaces */ .subspace-visual { display: flex; justify-content: center; align-items: center; gap: 10px; margin-top: 10px; } .circle { border-radius: 50%; display: flex; align-items: center; justify-content: center; font-size: 10px; font-weight: bold; backdrop-filter: blur(5px); border: 1px solid rgba(255,255,255,0.2); } .circle-past { width: 60px; height: 60px; background: rgba(99, 102, 241, 0.2); border-color: #6366f1; z-index: 1; } .circle-new { width: 50px; height: 50px; background: rgba(6, 182, 212, 0.2); border-color: #22d3ee; margin-left: -15px; z-index: 2; } .visual-label { text-align: center; font-size: 11px; color: #64748b; margin-top: 5px; } /* Results Section */ .stats-container { display: flex; justify-content: space-around; align-items: center; margin-top: 10px; } .stat-item { text-align: center; } .stat-value { font-size: 36px; font-weight: 800; background: linear-gradient(to bottom, #fff, #94a3b8); -webkit-background-clip: text; -webkit-text-fill-color: transparent; } .stat-desc { font-size: 12px; color: #64748b; text-transform: uppercase; margin-top: 4px; } .stat-plus { font-size: 24px; color: #22d3ee; vertical-align: super; } /* Footer */ .footer { margin-top: auto; border-top: 1px solid rgba(255, 255, 255, 0.1); padding-top: 15px; display: flex; justify-content: space-between; align-items: flex-end; font-size: 10px; color: #475569; } .source-link { color: #64748b; text-decoration: none; } .disclaimer { max-width: 70%; } /* Key Features List */ .features { list-style: none; display: grid; grid-template-columns: 1fr 1fr; gap: 10px; } .features li { display: flex; align-items: center; gap: 8px; font-size: 12px; color: #cbd5e1; } .features li::before { content: '✓'; color: #22d3ee; font-weight: bold; } ELLA Efficient Lifelong Learning for LLMs SOTA 2026 核心问题：灾难性遗忘大语言模型（LLM）在顺序学习新任务时，容易覆盖旧知识。传统的正交更新方法限制过于严苛，限制了知识的正向迁移。 ELLA 突破：引入选择性子空间去相关策略，通过轻量级正则化惩罚与旧任务关键方向的对齐，同时复用通用低能量子空间。技术原理 ELLA 通过各向异性收缩算子（Anisotropic Shrinkage Operator）限制干扰，实现稳定性与可塑性的平衡。 LELLA = || ΔWt ⊙ Wpast ||²F 正则化项惩罚新更新与历史高能量方向的对齐 Wpast ΔWt 选择性惩罚重叠子空间框架优势无需存储旧数据无需扩展参数规模内存占用减少 35× 增强零样本泛化能力计算开销极小适配 T5 / LLaMA 实验表现在多项基准测试中达到最先进（SOTA）性能，显著提升模型对新旧任务的兼顾能力。 9.6% 准确率提升 35× 内存占用减少声明：视频由NotebookLM自动生成，资料来源：ELLA: Efficient Lifelong Learning for Adapters in Large Language Models (arXiv:2601.02232) arxiv.org/pdf/2601.02232

核心问题：灾难性遗忘

大语言模型（LLM）在顺序学习新任务时，容易覆盖旧知识。传统的正交更新方法限制过于严苛，限制了知识的正向迁移。

ELLA 突破： 引入选择性子空间去相关策略，通过轻量级正则化惩罚与旧任务关键方向的对齐，同时复用通用低能量子空间。

技术原理

ELLA 通过各向异性收缩算子（Anisotropic Shrinkage Operator）限制干扰，实现稳定性与可塑性的平衡。

L_ELLA = || ΔW_t ⊙ W_past ||²_F

正则化项惩罚新更新与历史高能量方向的对齐

W_past

ΔW_t

选择性惩罚重叠子空间

框架优势

无需存储旧数据

无需扩展参数规模

内存占用减少 35×

增强零样本泛化能力

计算开销极小

适配 T5 / LLaMA

实验表现

在多项基准测试中达到最先进（SOTA）性能，显著提升模型对新旧任务的兼顾能力。

9.6%

准确率提升

35×

内存占用减少

ELLA Efficient Lifelong Learning for LLMs

🌟 智谱 GLM-5 已上线