SourceCheck is an experimental project presented on the Koala Chat open-source channel that tackles the core trust problem of LLM outputs: fluent, confident answers with no built-in way to verify them. Its guiding principle is simple — the LLM should generate citations pointing at answers, not just generate answers — building on the earlier reviewdeck project's "cite instead of copy" approach.
Key points
- The problem goes beyond hallucination. LLMs can distort real citations, selectively present evidence, use vague sourcing ("studies show…"), and amplify clickbait. A 2025 Columbia University study of 8 AI search tools reported error rates from 37% (Perplexity) to 94% (Grok 3), with models often citing wrong article versions or fabricating links. Confidence and correctness are two different things.
- Existing approaches fall short: post-hoc citation verifiers (SourceCheckup, llm-citation-verifier, SemanticCite, RefCheckAI) only check output after generation; RAG variants (Self-RAG, CRAG) still let models distort retrieved context; benchmarks (TrustLLM) compare systems but don't make outputs trustworthy.
- Fact check (clickbait detection): source-quality ratings (academic papers > official docs > authoritative media > self-media), title-vs-body consistency checks (e.g., a "coffee causes cancer" headline distorted from a WHO finding about hot beverages), and single-source vs. multi-source flags.
- Traceable PostgreSQL RAG: every claim is pinned to a specific doc version (PostgreSQL 16.2, section 8.14.2) with verified quotes and confidence; SQL examples link to the original documentation block; outdated sources are marked, and version mismatches (PG15 vs PG16) are warned about.
- Verification cost: precomputed embedding indexes, cached results, strict/fast modes.
- Unattributable knowledge: claims from pretraining are explicitly labeled "internal knowledge, not externally verifiable."
- UX friction: collapsed citations by default, color-coded trust levels, user-adjustable confidence thresholds.
- Adversarial fake sources: domain whitelists, multi-source cross-checks, community reporting.
- Koala Chat OSS course: https://koala-oss.app/course/
- SourceCheckup (Wu et al., 2024): https://github.com/kevinwu23/SourceCheckup
- llm-citation-verifier: https://github.com/DWFlanagan/llm-citation-verifier
- SemanticCite: https://github.com/sebhaan/semanticcite
- RefCheckAI: https://github.com/Sydney-Informatics-Hub/RefCheckAI
- OpenScholar: https://github.com/OpenScholar
The three-layer architecture
1. Protocol design — outputs are decomposed into atomic claims, each with references containing source URL, exact section, verification method, confidence score, and a checksum. Core principles: claims without references are treated as unverified; users can jump to the source in one click; sources can be independently verified; third parties can reproduce verification. 2. Reference localization + deterministic verification — precise anchoring via semantic search, document structure (headings, lists, tables), and version pinning. Verification methods: exact matching (LCS/edit distance), embedding-similarity checks (e.g., sentence-transformers cosine similarity), structural/logic checks (did the LLM turn correlation into causation?), and multi-source cross-validation. 3. Skill and UI SDK — a reusable skill that segments LLM output into claims, locates references, and verifies them; UI components include collapsible citation cards with ✅/⚠️/❌ status, a trust dashboard, and interactive re-verification.
Demo scenarios
Limitations and mitigations
Conclusion
SourceCheck proposes a paradigm shift from "black-box generation" to a structured claims-and-citations pipeline — importing academic citation norms into LLM output via a protocol + SDK + ecosystem rather than a single tool. The author argues that for AI to land in healthcare, law, finance, and research, verifiability is not optional but mandatory: *the value of AI output lies not just in what it says, but in how much you can verify it.*