Astronomy x Graphics: Erase the Galaxy and the Classifier Still Gets 3 out of 4 Right

Forum topic · QianXun · 2026-11-05

Summary

A re-evaluation study posted on arXiv (2611.06484, Eraldo Pereira Marinho) tests how much of the reported accuracy in small-data astronomical image classification actually comes from object morphology. The AstroClassifier dataset contains 370 catalog-backed physical objects in three DSS FITS bands, split by object into 222/75/73 for a four-class task (galaxies, globular clusters, nebulae, open clusters), with balanced accuracy and a 0.25 chance baseline. Full-field adaptive views scored 0.773-0.874 across a custom CNN-Transformer, ResNet-50, and ViT-B/16. Masking the central 128x128 pixels dropped scores only to 0.731-0.800, and logistic regression or shallow decision trees fed just acquisition records and catalog attributes reached 0.721-0.735 with no images at all. The results show substantial class information resides in field-of-view and acquisition metadata rather than the target object. The paper also notes architecture comparisons are confounded by pretraining, and no architecture difference survived Holm correction.

Key points

A re-evaluation of small-data astronomical image classifiers, arXiv:2611.06484 by Eraldo Pereira Marinho (17 pages, 3 figures, astro-ph.IM + cs.CV), asks how much of the reported high scores comes from object morphology, versus object-independent and acquisition shortcuts.

Dataset and protocol

AstroClassifier: 370 catalog-backed physical objects, each represented by three Digitized Sky Survey (DSS) FITS bands; four classes: galaxies, globular clusters, nebulae, open clusters.
Split by physical object: 222 train / 75 validation / 73 test, so the same object never appears on both sides of the split. Balanced accuracy used; random-guess baseline is 0.25.
Three fixed seeds, one-shot object-independent testing; all predictions written before labels are opened. Architecture comparisons underwent Holm correction.
Three architectures: a from-scratch custom CNN-Transformer, ImageNet-pretrained ResNet-50, and pretrained ViT-B/16. Three representations: adaptive full-field, common angular field, and two central-masking controls.

Scores

| Reading | Balanced accuracy | What the model saw | |---|---|---| | Adaptive full-field, custom CNN-Transformer | 0.773 ± 0.014 | Full image | | Adaptive full-field, ResNet-50 | 0.874 ± 0.054 | Full image | | Adaptive full-field, ViT-B/16 | 0.866 ± 0.023 | Full image | | Center-masked (128×128 pixels zeroed) adaptive full-field | 0.731 to 0.800 | Sky outside the center | | Acquisition records + catalog attributes only (logistic regression / depth-3 decision tree) | 0.721 to 0.735 | No image at all | | Random guess | 0.25 | No information |

Interpretation

Going from ~0.87 (full images) to ~0.72 (metadata only) costs only about 0.15; the real cliff is from 0.72 down to 0.25. A substantial share of discriminative information is not on the central object.
A secondary source (Pebblous blog) reports that knowing the class alone explains ~0.58 of the variance in exposure field width — different classes are observed with different exposure times, filters, and field sizes, so "how wide is this image" itself carries class labels. The authors call this an acquisition shortcut.
Caveats the paper itself states: the architecture comparison changes architecture and pretraining simultaneously, so the 0.101 gap cannot be attributed to either variable. ResNet-50 vs ViT-B/16 differ by only 0.007 (95% clustered bootstrap interval -0.048 to 0.062, contains zero); no main architecture comparison rejects the null after Holm correction.
The authors do not claim classifiers learned nothing — they state that within this range, high accuracy cannot be interpreted as evidence the classifier learned target morphology. The masked/no-image controls show the information suffices, but do not alone prove which shortcut a model uses; intervention experiments would be needed for that.

Recommended evaluation practices

Split by physical object, normalize field-of-view and acquisition attributes, run masking and no-image controls, treat architecture and pretraining as separate variables, and never cherry-pick the best of several test runs.

Sources

arXiv:2611.06484, "Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts", Eraldo Pereira Marinho
Pebblous Blog, "An Astronomical Image Classifier Still Gets Three in Four with the Galaxy Erased" (secondary interpretation)