Summary
GRAC (Graph-Based Retrieval-Augmented Captioning) is a framework for automated surgical scene captioning proposed by Rochak Dhakal, Chen Zhao, Soup Cobb, Xin Qian, and Xiaofeng Yang (arXiv 2611.09866). It represents surgical scenes as graphs capturing relationships among instruments, actions, and anatomical structures. Given an input scene graph, GRAC retrieves relevant graph-caption pairs from training data, compares instrument-tissue interactions between retrieved examples and the input scene, and uses comparison features to condition caption generation. On the EndoVis-18 benchmark, GRAC achieves a CIDEr-D of 5.88, outperforming SGT (4.23) and the strongest conventional Transformer baseline (3.36), and reaches 52.6% triplet-set exact accuracy for instrument-action-target triplets versus 32.7% for SGT and 26.2% for the strongest conventional baseline. On TORS, GRAC attains a CIDEr-D of 5.06 versus 3.80 for the strongest baseline. The results indicate that combining structured scene representations with retrieval-augmented examples yields more accurate and contextually meaningful surgical reports.
Paper Overview
- Field: Computer Vision (CV)
- Authors: Rochak Dhakal, Chen Zhao, Soup Cobb, Xin Qian, Xiaofeng Yang
- Published: 2026-11-06
- arXiv: 2611.09866
Abstract
Purpose: Surgical operative reports support procedural documentation, surgical training, performance assessment, and workflow review. However, extracting meaningful information often requires extensive manual review and clinical expertise. Automated surgical scene descriptions could reduce this burden by leveraging recurring surgical activities and prior knowledge.
Methods: The authors propose GRAC (Graph-Based Retrieval-Augmented Captioning), a framework that represents surgical scenes as graphs capturing relationships among instruments, actions, and anatomical structures. Given an input scene graph, GRAC learns to retrieve relevant graph-caption pairs from training data. It compares instrument-tissue interactions between retrieved examples and the input scene, using comparison features to condition the representation of retrieved captions during generation.
Results:
- On EndoVis-18, GRAC achieves a CIDEr-D of 5.88 (SGT: 4.23; strongest conventional Transformer baseline: 3.36).
- Instrument-action-target triplet set exact accuracy: 52.6% (SGT: 32.7%; strongest conventional baseline: 26.2%).
- On TORS, GRAC achieves a CIDEr-D of 5.06 (strongest baseline: 3.80).
Conclusion: Combining structured surgical scene representations with retrieved examples improves surgical scene captioning and produces more contextually meaningful descriptions.
*Auto-collected on 2026-11-10.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636983