νGPT: Fixed-Point Attention Brings Million-Token Context to Consumer GPUs

Forum topic · QianXun · 2026-05-15

Summary

νGPT (nu-GPT) is a new transformer architecture built around fixed-point attention, aimed at solving the quadratic compute and exploding KV-cache memory costs of standard attention in long-context language models. The core idea is a nonlinear ν-transformation operator: instead of explicitly storing every key-value pair, νGPT iteratively solves for the fixed point that the attention distribution converges to. As a result, memory usage plateaus at a lightweight level instead of growing linearly with context length. According to the post, νGPT reportedly processed over 2 million tokens with coherent logic, ran inference 3-5x faster than comparable long-context transformers, and enabled workloads that previously required A100 clusters to run on consumer cards such as the RTX 5090. The author frames this as evidence that elegant architectural and mathematical refinements can outweigh brute-force scaling, marking a shift from heavy compute toward algorithmic efficiency in AI.

Introduction

If you were reading a 1-million-word novel, could you still clearly remember a minor detail from page one when you reached the last page? That is hard for humans — and it used to be a memory nightmare for large language models. Because the compute cost of standard attention scales quadratically with sequence length, handling million-token contexts typically requires clusters of GPUs. νGPT claims to change this by finding a "fixed point" of attention.

1. Why memory always runs out

In the standard Transformer, every token must attend to all previous tokens:

Double the length, quadruple the compute.
The KV cache keeps growing until it exhausts GPU memory, breaking inference.

Previous fixes — sliding windows or sparsification — are like only remembering the last few pages of a book, causing models to lose track in complex reasoning.

2. νGPT: a fixed point for attention

νGPT introduces a new mathematical primitive: fixed-point attention.

ν-transformation operator: a nonlinear operator, denoted ν. The researchers observed that as models process sequences, the attention distribution tends toward a mathematical equilibrium.
Iterative solving instead of explicit storage: if attention converges to a fixed point, there is no need to memorize every key-value pair. νGPT uses an efficient iterative algorithm to solve directly for that fixed-point distribution.
Constant memory: in certain modes, νGPT's memory cost for context no longer grows with length; it stays pinned at a very lightweight level.

Feynman-style analogy: conventional attention is like a perfectionist who rereads every earlier note each time a new word arrives. νGPT is like a memory master who maintains a dynamic "mental outline" in their head — each new word only requires a small update to that outline. Once the outline reaches its stable fixed point, no matter how many words follow, the load no longer increases.

3. Results: million-token contexts on a single consumer GPU

Per the post's reported benchmarks:

Unbounded length: successfully processed over 2 million tokens with coherent, self-consistent logic.
Speed: inference is 3-5x faster than a comparable long-context Transformer.
Hardware democratization: workloads that previously required an A100 cluster now run smoothly on consumer GPUs such as the RTX 5090.

Editor's note

νGPT reinforces a familiar lesson: small architectural improvements can beat orders of magnitude of brute-force compute. By re-examining the essence of attention from a mathematical standpoint — as a search for equilibrium — the heavy computational burden dissolves. The fixed-point idea signals a move from the era of brute-force computing toward one of elegant algorithms.

*Note: this article is based on a 2026 long-context architecture research write-up as posted on the forum.*