Paper Overview
Field: Machine Learning Authors: Varun Reddy, Bernardo L. Sabatini, Houman Safaai Published: 2026-09-25 arXiv: 2609.31589
Summary
Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivities across 48 settings.
Key Findings
- On MNIST, class decodability largely survives the collapse, but readout learning remains slow with a fixed learning rate; Adam learns faster despite deeper collapse.
- Calibrating the baseline readout to class priors suppresses collapse and accelerates learning.
- Weaker feedback trades slower learning for less collapse.
- Replacing the error with its sign sustains collapse; subtracting the batch mean of the signal prevents persistent collapse and improves learning on test settings.
- Similar effects appear in deeper networks, convolutional networks, and CIFAR-10, with severity and cost depending on the readout, optimizer, and input statistics.
*Auto-collected on 2026-09-29*