Beyond Monolithic LLMs: A Paradigm Shift from All-Purpose Giant Models to a Society of Experts
This post discusses an alternative path for generative AI, arguing that the dominant "scale-is-everything" approach is hitting physical, economic, and cognitive limits — and proposing a shift toward Domain-Specific Superintelligence (DSS) organized as a *society of experts*.
1. The Energy Crisis and the "Reversal Curse": The Cost of Brute-Force Scaling
- Training frontier models is energy-intensive, but inference-stage energy consumption is now the bigger concern as models become high-traffic products.
- Reasoning models, which generate long internal chains of thought per query, amplify single-query compute and energy costs by 70–100x.
- Grid impact: Virginia (a global data-center hub) projects summer peak load 70% higher by 2045 vs. 2022, driven almost entirely by data centers, prompting new gas peaker plants that conflict with clean-energy goals.
- Water footprint: a large data center may evaporate 1.1 million gallons of water per day for cooling (roughly a 10,000-person town's usage); indirect water use by power plants is typically 3–4x direct cooling use. A single query to a large model (e.g., Gemini) consumes about 0.26 mL of water on average — billions of queries add up to billions of liters annually.
- Capability flaw: the "Reversal Curse" — a model trained on "A is B" often cannot infer "B is A" — shows that statistical pattern-matching lacks genuine abstraction and symmetric reasoning.
- Decoupling capability from scale: a specialized 7B medical model can outperform a general 175B model in its domain.
- Robustness and diversity: multiple experts cover each other's failures and reduce single-model bias.
- Efficiency and composability: modular, "Lego-style" expansion; small models can run on edge devices for local inference.
- Interpretability and auditability: training grounded in explicit symbolic abstractions makes decisions traceable — critical for healthcare, law, and finance.
2. Neurosymbolic AI: Filling the Missing Abstraction
Current LLMs learn statistical patterns but skip the construction of abstract concepts and rules. Neurosymbolic AI combines neural networks' perception and generalization with symbolic systems' interpretable, verifiable reasoning, achieving more with less data and compute. It aims to make models not just "know that" but "know why."
3. Graph Algebra and GraphMERT: Knowledge Graphs as Computable Coordinates
The proposed technical route: first build explicit symbolic abstractions (knowledge graphs, ontologies, formal logic), then use them to generate high-quality training curricula for small models.
GraphMERT (Graphical Multidirectional Encoder from Transformers) is a small encoder-only transformer that distills reliable knowledge graphs from raw text in three steps:
1. Unified representation: map domain text and seed knowledge graphs into a shared representation space. 2. Training and prediction: learn correspondences between text context and graph structure, then predict new semantic completions (new triples). 3. LLM-assisted cleaning: use a large LLM as an auxiliary to polish and filter predicted triples against the source text.
Its key guarantees are factuality (every triple traceable to source text) and validity (semantics consistent with domain ontology). On diabetes literature, the 80M-parameter GraphMERT achieved a factual FActScore of 69.8%, versus 40.2% for a 32B baseline LLM, and scored ~26 points higher on semantic validity.
Beyond that, graph algebra embeds knowledge graphs into geometric algebra spaces where logical reasoning becomes geometric transformation — enabling computable counterfactual reasoning (e.g., the CFKGR task, addressed via methods like COULDD, where KG embeddings detect which facts should change under a hypothesis and which should remain fixed).
4. The DSS Society of Experts: Small Models Beating Trillion-Parameter Giants
Instead of one universal giant model, DSS envisions many small, deeply specialized expert models coordinated by an orchestrator that decomposes tasks and routes them. Advantages:
5. Conclusion: Toward a Sustainable AI Future
The alternative trajectory — neurosymbolic abstraction, knowledge graphs as computable coordinates, hallucination-free KG construction via GraphMERT, and a DSS society of experts — aims to align algorithmic progress with physical constraints, making generative AI a sustainable force for economic empowerment and democratized knowledge rather than an environmental burden.