Predictive Coding and Backpropagation: Equivalence, Biology, and Algorithms

1. Introduction

Core Question

How does the brain achieve efficient learning without a backpropagation algorithm? Predictive coding theory offers an elegant answer: through free energy minimization and hierarchical prediction error propagation, the brain mathematically replicates backpropagation's computational effects using fully biologically plausible mechanisms.

Predictive coding was originally proposed by Rao and Ballard (1999) as a theoretical framework for visual cortex computation. Its core idea is that the brain does not passively process sensory input but actively predicts inputs and updates its internal model through prediction errors. Friston later developed this into the free energy principle, creating a unified framework for understanding brain cognition.

Meanwhile, backpropagation—the cornerstone of modern deep learning—has long faced criticism for its biological implausibility. The weight transport problem, global coordination requirements, and forward-backward separation all conflict with biological neural system constraints. Predictive coding networks provide a new solution to this dilemma.

Timeline of PC-BP Equivalence Discoveries

| Year | Researchers | Core Contribution | |------|-------------|-------------------| | 1999 | Rao & Ballard | Neural network implementation of predictive coding | | 2017 | Whittington & Bogacz | Rigorous proof of PC-BP equivalence | | 2020 | Millidge et al. | Relaxed-constraint predictive coding model | | 2022 | Song et al. | Backprop approximation for arbitrary computational graphs | | 2024 | Salvatori et al. | Stable and fast automated learning algorithm |

2. Theoretical Framework

2.1 Fundamentals of Predictive Coding

Predictive coding is grounded in the free energy principle, systematically expounded by Bogacz (2017). The variational free energy is defined as:

This framework unifies perception and learning: perception corresponds to minimizing free energy with respect to hidden states, while learning corresponds to minimizing free energy with respect to model parameters. Friston (2018) further emphasized the normative character of this framework—it describes not only how the brain works, but why it works that way.

2.2 The Biological Predicament of Backpropagation

Lillicrap et al. (2020) systematically analyzed the compatibility between backpropagation and brain computation, identifying three core problems:

• Weight Transport Problem: Backpropagation requires precise transposition of forward weights, which is biologically difficult to implement
• Global Coordination: Requires globally synchronized forward-backward passes, contradicting the brain's asynchronous nature
• Memory Overhead: Requires storage of all intermediate activations; memory requirements scale with depth

2.3 Computational Architecture of Predictive Coding

The predictive coding architecture proposed by Rao & Ballard (1999) exhibits encoding-decoding symmetry: each layer in the cortical hierarchy performs both encoding and decoding functions. Hierarchical prediction error propagation implements local credit assignment:

\[\varepsilon_l = x_l - f(W_{l+1} x_{l+1})\]
\[\tau \dot{x}_l = \varepsilon_l - f'(x_l) \varepsilon_{l-1}\]

The review by Keller & Mrsic-Flogel (2018) shows that this architecture corresponds closely to cortical microcircuit anatomy: different dendritic compartments of pyramidal neurons may separately encode predictions and prediction errors, while inhibitory interneurons may perform precision regulation.

3. Mathematical Equivalence and Convergence

3.1 Rigorous Proof of PC-BP Equivalence

Core Finding: Whittington & Bogacz (2017) first rigorously proved that when the update ratio parameter γ = 0.5, the weight updates in a predictive coding network are completely equivalent to the backpropagation algorithm.

This equivalence proof involves perturbation analysis of coupled dynamical systems. Let neural activity update on timescale τ, and weights update on timescale τ/γ. When γ = 0.5, the effective error signal in predictive coding satisfies:

\[\tilde{\delta}_l = \varepsilon_l \cdot f'(x_l)\]
\[\tilde{\delta}_l = W_{l+1}^T \tilde{\delta}_{l+1} \cdot f'(x_l)\]

This is entirely consistent with backpropagation's error signal propagation. Millidge et al. (2022c) further proved that for any γ > 0, predictive coding asymptotically converges to backpropagation at convergence rate O(γ).

3.2 Extension to Arbitrary Computational Graphs

Marino (2021) and Millidge et al. (2022c) extended PC-BP equivalence to arbitrary computational graphs. For any directed acyclic graph, generalized predictive coding networks are defined by:

\[\varepsilon_i = x_i - f_i(\{x_j\}_{j \in pa(i)})\]
\[\mathcal{F} = \sum_i \frac{1}{2} \varepsilon_i^T \Sigma_i^{-1} \varepsilon_i\]

where pa(i) denotes the parent nodes of node i. Free energy minimization is equivalent to reverse-mode automatic differentiation for this graph. This result provides a theoretical foundation for the "Turing completeness" of predictive coding.

3.3 Unified View from Variational Inference

Marino (2021) revealed the mathematical isomorphism between predictive coding and variational autoencoders (VAEs). The VAE evidence lower bound:

\[\mathcal{L}_{ELBO} = \mathbb{E}_{q_\phi(z|x)}[\ln p_\theta(x|z)] - D_{KL}[q_\phi(z|x) \| p(z)]\]

corresponds to predictive coding's free energy. The key difference is that VAE inference is "amortized" (one-shot encoding), while predictive coding inference is "iterative" (dynamical convergence). This correspondence provides a theoretical basis for hybrid designs combining both methods.

4. Biological Plausibility

4.1 Neural Implementation of Local Plasticity Rules

The predictive coding learning rule has the basic form of Hebbian plasticity, augmented with prediction error modulation:

\[\Delta W_{ij} \propto \varepsilon_i \cdot f'(x_i) \cdot x_j\]

Whittington & Bogacz (2017) emphasize that this rule depends only on locally available information and is highly compatible with known synaptic plasticity mechanisms. ε_i can be encoded by deviations in membrane potential, f'(x_i) can be implemented by spiking threshold properties, and x_j is presynaptic activity.

Biological Implementation Advantages:

• No global error signal required
• No weight transport required
• No precise temporal synchronization required
• Supports online learning

4.2 Cortical Microcircuit Structural Correspondence

The review by Keller & Mrsic-Flogel (2018) evaluates neurophysiological evidence for predictive coding:

Supporting Evidence:

• Membrane potential distribution of pyramidal neurons consistent with prediction-error coding
• Repetition suppression effects in visual cortex for familiar stimuli
• Top-down modulation effects
• Neuromodulatory mechanisms for precision regulation

Open Questions:

• Precise separation of prediction and error encoding
• Specific computational roles of inhibitory neurons
• Functional differences across cortical layers
• Network construction during development

4.3 Neurophysiological Experimental Evidence

The classical work by Rao & Ballard (1999) used predictive coding to explain various "non-classical receptive field" effects. Recent work by Song et al. (2024) identified neurons specifically encoding prediction errors that selectively respond to "expectation violations," supporting the core hypothesis of predictive coding. However, interpretation disagreements remain regarding the precise computational roles of these neurons.

5. Algorithmic Innovations and Extended Frameworks

5.1 Inference Learning Paradigm

Song et al. (2024) proposed "inference learning," a methodological breakthrough that fundamentally addresses the practical deployment challenges of predictive coding networks. Inference learning transforms the iterative inference process into direct, single-step computations while preserving biological plausibility.

Key contributions include:

• Direct Inference Mapping: Learning to approximate iterative inference dynamics with feedforward mappings
• Inference-Only Learning: Enabling training without backpropagation through the inference loop
• Biological Compatibility: Maintaining local learning rules while achieving competitive performance

This paradigm demonstrates that predictive coding principles can scale to modern deep learning architectures without sacrificing the biological advantages that motivate the framework.

Key Points

• Mathematical Equivalence: Predictive coding networks can exactly replicate backpropagation gradient computation when update ratio γ = 0.5, proven by Whittington & Bogacz (2017)
• Convergence Guarantee: For any γ > 0, predictive coding asymptotically converges to backpropagation at rate O(γ)
• Biological Plausibility: PC learning rules require only local information (postsynaptic error, presynaptic activity, activation derivative), avoiding the weight transport problem
• Computational Graph Generality: PC-BP equivalence extends to arbitrary directed acyclic graphs, providing Turing-complete learning capabilities
• Unification with VAEs: Predictive coding's free energy corresponds to the VAE evidence lower bound, differing primarily in amortized vs. iterative inference
• Open Biological Questions: Precise neural mechanisms for prediction vs. error encoding, inhibitory neuron roles, and developmental network formation remain unresolved
• Algorithmic Scalability: Inference learning (Song et al., 2024) addresses practical deployment challenges while preserving biological advantages

References

• Bogacz, R. (2017). https://doi.org/10.1016/j.jmp.2015.11
• Friston, K. (2018). https://doi.org/10.1038/s41593-018-02
• Keller, G. B., & Mrsic-Flogel, T. D. (2018). https://doi.org/10.1016/j.neuron.2018
• Lillicrap, T. P., et al. (2020). https://doi.org/10.1038/s41583-020-02
• Marino, J. (2021). https://doi.org/10.48550/arXiv.2011.0
• Millidge, B., et al. (2022c). https://doi.org/10.1162/neco_a_01497
• Rao, R. P. N., & Ballard, D. H. (1999). https://doi.org/10.1038/4580
• Salvatori, T., et al. (2024). https://doi.org/10.1038/s41593-023-01
• Song, Y., et al. (2024). https://doi.org/10.1038/s41593-023-01
• Whittington, J. C. R., & Bogacz, R. (2017). https://doi.org/10.1162/NECO_a_00949