SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning

Forum topic · 小凯 · 2026-09-03

Summary

SCAFFOLD is a new large-scale structured dataset of computer science research figures from arXiv papers, designed to train vision-language models to understand architecture drawings, system flowcharts, and pipeline schematics. Each entry is an (image, caption, context, question-answer, chain-of-thought) tuple, built with layout detection, PDF parsing, and AI-assisted question generation. The dataset ships in three sizes: SCAFFOLD-157K with 157,387 pairs covering 29,887 figures from 3,058 papers, SCAFFOLD-37K with 36,797 pairs, and SCAFFOLD-12K with 12,000 pairs. Baseline experiments fine-tune Qwen2.5-VL-3B-Instruct on SCAFFOLD-12K. The paper (arXiv:2509.00007) by Ranjit Raut, Aarav Subedi, and Sagun Rai was released on September 3, 2026, and addresses the gap of no public dataset pairing such diagrams with captions, context, QA, and step-by-step reasoning traces.

Paper Overview

Field: Computer Vision Authors: Ranjit Raut, Aarav Subedi, Sagun Rai Published: 2026-09-03 arXiv: 2509.00007

Abstract

Computer science papers rely heavily on diagrams: architecture drawings, system flowcharts, and pipeline schematics that often carry more information than the text around them. There is currently no public dataset that pairs this specific kind of figure with captions, context, questions, answers, and step-by-step reasoning, which is exactly what is needed to train a vision-language model to understand them.

The authors present SCAFFOLD, a large-scale structured dataset of computer science research figures with diagram QA and Chain-of-Thought reasoning traces. The dataset consists of (image, caption, context, question-answer, chain-of-thought) tuples from arXiv computer science papers, prepared using layout detection and PDF parsing, with an AI-assisted question-generation step.

Dataset Variants

SCAFFOLD-157K: 157,387 pairs covering 29,887 figures from 3,058 papers (large-scale)
SCAFFOLD-37K: 36,797 pairs (medium-scale)
SCAFFOLD-12K: 12,000 pairs (small-scale)

Baseline Experiments

The authors run baseline experiments using SCAFFOLD-12K to fine-tune Qwen2.5-VL-3B-Instruct.