1 Introduction to Machine Learning and Deep Learning

The Preface argued that quantitative economics needs new tools because the models of interest, heterogeneous-agent economies, OLG models with aggregate risk, integrated assessment models, all share state spaces too large for traditional grid-based methods. This chapter takes that argument as given and supplies the technical machinery the rest of the manuscript will build on. Readers who want a fuller pitch for why deep learning is the right response should re-read the Preface; readers who already accept the premise can dive directly into the machinery below.

We begin with a brief refresher on the motivation, mostly to fix vocabulary and citations, then survey the three broad paradigms of machine learning (supervised, unsupervised, and reinforcement learning), and then develop the core technical machinery: neural network architectures, loss functions, gradient-based optimization, backpropagation, weight initialization, activation functions, and the modern theory of generalization including the double descent phenomenon. Readers already comfortable with these topics may skim this chapter and proceed to Chapter Chapter 2, where the economic applications begin.

The foundational references for the material in this chapter include McCulloch & Pitts (1943), Hebb (1949) and Rosenblatt (1958) for the historical origins of artificial neurons and the first learning rules, Cybenko (1989) and Hornik et al. (1989) for universal approximation, Rumelhart et al. (1986) for backpropagation, Robbins & Monro (1951) for the stochastic-approximation roots of SGD, Geman et al. (1992) for the bias/variance dilemma that underpins modern generalization theory, Kingma & Ba (2015) for the Adam optimizer, Ioffe & Szegedy (2015) and Srivastava et al. (2014) for batch normalization and dropout, and Goodfellow et al. (2016) for a comprehensive textbook treatment, complemented by the historical survey of Schmidhuber (2015).

1.1Motivation and Applications¶

The past decade has witnessed a remarkable convergence of three developments that have transformed machine learning from a niche academic pursuit into a practical tool of extraordinary power: the availability of large-scale datasets, the advent of massively parallel hardware (GPUs and TPUs), and algorithmic innovations in training deep neural networks (Figure Figure 1.1). While much of the public attention has focused on applications such as image recognition, natural language processing, and game playing, the implications for economics and finance are equally profound.

Deep learning has already demonstrated its potential across a broad range of economic applications. In macroeconomics, neural networks serve as global approximators of policy and value functions in high-dimensional equilibria that classical grid-based methods cannot reach Maliar et al., 2021Azinovic et al., 2022; heterogeneous-agent extensions encode cross-sectional distributions via histograms or permutation-invariant moment networks Young, 2010Han et al., 2024Yang et al., 2025. Search-and-matching models with aggregate shocks Payne et al., 2025 and continuous-time macro-finance settings requiring HJB approximation or deep-BSDE solvers Gopalakrishna, 2024Duarte et al., 2024Han et al., 2018 add further coverage, as do optimal monetary policy rules under persistent supply shocks Nuño et al., 2024. In climate economics, surrogate-based workflows solve integrated assessment models and derive Pareto-improving carbon-tax rules in OLG--IAMs with deep uncertainty Kübler et al., 2026Folini et al., 2025Fernández-Villaverde et al., 2025. In finance, surrogate models accelerate option pricing, sovereign-default computation, and portfolio optimization Hutchinson et al., 1994Scheidegger & Treccani, 2018Arellano, 2008Gaegauf et al., 2023Chen et al., 2025. For structural estimation, neural-network surrogates make inference tractable and enable global uncertainty quantification for IAMs Kase et al., 2022Friedl et al., 2023Chen et al., 2026. Finally, these methods connect to an earlier generation of neural-network function approximation in economic computation: adaptive learning Chen & White, 1998, derivative pricing Hutchinson et al., 1994, and parameterized expectations Duffy & McNelis, 2001, with Valaitis & Villa (2024) showing how the parameterized-expectations approach extends to contemporary deep architectures; structural discrete-choice estimation rounds out the historical picture Norets, 2012.

Two themes cut across these application areas and motivate the rest of the script. First, every application area listed there involves a state space whose dimension grows with the number of agents, assets, shocks, or climate states; tensor-product grids become infeasible long before the modeling questions become uninteresting. Second, neural networks are universal approximators with cost that scales with parameter count rather than with $N^d$, so they are the natural replacement function class once the problem becomes high-dimensional. Subsequent chapters take this through-line and develop it: Chapter Chapter 2 introduces the DEQN methodology that all macro / heterogeneous-agent / search applications above share; Chapter Chapter 3 scales it to a 100+-country benchmark; Chapters Chapter 7--Chapter 8 develop the continuous-time analogue; Chapter Chapter 9 develops the Gaussian-process methodology, and Chapters Chapter 10--Chapter 11 put the deep-surrogate and GP machinery to work on structural estimation and integrated assessment models.

In this course, we focus on the recent advances enabled by deep neural networks, modern hardware, and algorithmic innovations in training.

For central banks, in particular, these tools address pressing practical needs. Many of the models listed above, such as heterogeneous-agent New Keynesian (HANK) models, search and matching models with aggregate shocks, overlapping-generations (OLG) economies with aggregate risk, and integrated assessment models coupling climate and economic dynamics, involve state spaces of dimension $d \gg 5$, where traditional grid-based numerical methods are computationally infeasible: a tensor-product grid with $N$ points per dimension requires $N^{d}$ nodes, the canonical curse of dimensionality of Bellman (1961). Deep learning provides a mesh-free function approximator. For Barron-class functions, two-layer networks achieve dimension-independent rates in the number of hidden units (often stated as squared $L^2$ error of order $\mathcal{O}(1/n_1)$ under a bounded Barron norm), whereas tensor-product grids for generic smooth functions suffer rates such as $\mathcal{O}(M^{-2/d})$ in the total number $M=N^d$ of grid nodes. The point is not that every economic object satisfies a Barron bound, but that neural approximators avoid the mechanical tensor-product explosion. Chapter Chapter 2 returns to this comparison with concrete numbers for a six-shock DSGE.

Specifically, this course focuses on using deep neural networks as computational tools for: (i) solving high-dimensional dynamic stochastic general equilibrium (DSGE) models, (ii) approximating value and policy functions in continuous-time settings via physics-informed neural networks, (iii) constructing fast and accurate surrogate models for parameter estimation and uncertainty quantification, and (iv) leveraging Gaussian processes for sample-efficient Bayesian active learning Azinovic et al., 2022Friedl et al., 2023Chen et al., 2026. Supporting techniques include neural architecture search and adaptive multi-objective loss balancing (one option, ReLoBRaLo Bischof & Kraus, 2025, is developed in the notebooks; alternatives include SoftAdapt and GradNorm). For a comprehensive textbook treatment of the foundations, we refer the reader to Goodfellow et al. (2016) and Chollet (2017); for concise surveys we recommend LeCun et al. (2015).

1.2Types of Machine Learning¶

Before diving into the technical details, it is useful to recall the three broad paradigms of machine learning, each defined by the nature of the available data and the feedback signal provided to the algorithm.

1.2.1Supervised Learning¶

Given a set of labeled input--output pairs $\{(\x^{(i)}, y^{(i)})\}_{i=1}^{m}$, the goal is to learn a mapping $h:\mathcal{X}\to\mathcal{Y}$ that generalizes to unseen inputs. The two main tasks are regression and classification.

1.2.1.0.1Regression¶

($y \in \R$): predict a continuous target from input features. A simple linear model takes the form

where the parameters $\bm{\theta} = (\theta_0, \theta_1)$ are learned from data. Figure Figure 1.2 illustrates regression on a house-price dataset: each dot is a training observation, and the line is the fitted model.

Classification. ($y \in \{0,1,\dots,K\}$): assign an input $\x$ to one of $K$ discrete categories. A linear classifier predicts class 1 whenever $\w^\top \x + b > 0$ and class 0 otherwise. Figure Figure 1.3 shows a credit-scoring example: applicants are classified as low-risk or high-risk based on income and savings, and the dashed line is the learned decision boundary.

1.2.2Unsupervised Learning¶

Given only unlabeled data $\{\x^{(i)}\}_{i=1}^{m}$, the goal is to discover hidden structure without any target signal. Two common tasks are:

• Clustering: partitioning data into groups of similar observations. Example: segmenting firms into peer groups based on financial characteristics such as size, leverage, and profitability.

• Dimensionality reduction: compressing features into fewer dimensions while preserving important variation. Example: principal component analysis of yield curves, where three factors (level, slope, curvature) capture most of the cross-sectional variation.

Figure Figure 1.4 illustrates a clustering task: unlabeled data points in two dimensions are partitioned into three clusters, each indicated by a different color and centroid marker.

1.2.3Reinforcement Learning¶

In reinforcement learning, an agent interacts with an environment over a sequence of time steps. At each step $t$, the agent observes a state $s_t$, selects an action $a_t = \pi(s_t)$ according to its policy $\pi$, and receives a reward $r_t$ from the environment. The goal is to learn a policy that maximizes the expected cumulative discounted return:

where $\mathbb{E}_{\pi}[\,\cdot\,]$ is taken over the trajectory distribution induced jointly by the policy $\pi$ and the (possibly stochastic) environment dynamics, starting from a given initial-state distribution. Figure Figure 1.5 illustrates this agent--environment interaction loop.

Example: an algorithmic trader learning an execution strategy by optimizing realized profit over sequences of order placements; or a central bank learning an interest-rate rule by maximizing a welfare criterion over simulated macroeconomic trajectories.

1.3Course Focus: Supervised vs. Unsupervised Learning¶

This course begins with the supervised learning paradigm, which provides the essential building blocks: choosing a parameterized model, defining a loss function, and minimizing it via gradient descent.

The core methods of this course, DEQNs and PINNs, are not supervised in the classical labeled-data sense. More precisely, they are self-supervised residual methods: the economic equilibrium conditions or governing equations generate the training signal. To see why, recall the key distinction: in supervised learning, the loss function measures the discrepancy between the network’s prediction $\hat{y}^{(i)}$ and a known target label $y^{(i)}$, for example, a mean squared error $\frac{1}{m}\sum_i (\hat{y}^{(i)} - y^{(i)})^2$. This requires a dataset of correct input--output pairs.

In DEQNs and PINNs, no such labels exist. Consider the key differences:

• DEQNs: A neural network approximates the unknown policy function of a dynamic economic model. The loss function is the Euler equation residual, which measures how much the network’s output violates the model’s optimality conditions at sampled state points. The “correct” policy values are never provided; instead, the network discovers the equilibrium by driving these residuals to zero.

• PINNs: A neural network approximates the unknown solution to a partial differential equation (PDE). The loss function is the PDE residual, which evaluates how well the network satisfies the differential equation at randomly sampled collocation points. Again, the true solution is not available as training data; the network learns it by enforcing the PDE constraint.

In both cases, the training data consists only of input locations (sampled states or collocation points) with no associated output labels. The loss is defined entirely by the structure of the economic model or the governing equation, not by example solutions. They are therefore unsupervised in the narrow sense of using no labels, but the more informative term is equation-based self-supervision.

Despite this fundamental difference, the optimization machinery is shared: these approaches define a loss $J(\bm{\theta})$ over trainable parameters and minimize it via (stochastic) gradient descent. This is why we introduce the supervised learning pipeline first in the next section: it establishes the model--loss--optimizer framework that DEQNs and PINNs then adapt by replacing the data-driven loss with a physics-based one.

1.4The Supervised Learning Pipeline¶

Every supervised learning algorithm follows the same three-step recipe, regardless of whether the model is a linear regression, a random forest, or a deep neural network (Figure Figure 1.6). Understanding this pipeline is essential because the DEQN and PINN methods in later chapters modify step 2 (replacing data-driven losses with physics-based residuals) while keeping steps 1 and 3 intact.

Given a training set $\{(\x^{(i)}, y^{(i)})\}_{i=1}^{m}$ of input--output pairs, we seek a hypothesis $h:\mathcal{X}\to\mathcal{Y}$ parameterized by $\bm{\theta}$ that minimizes the empirical risk. For regression problems, the default choice is the mean squared error (MSE):

This loss is not chosen arbitrarily. If the data are generated by

then the log-likelihood of the sample is, up to constants, proportional to $-\sum_i (h_{\bm{\theta}}(\x^{(i)}) - y^{(i)})^2$. Minimizing MSE is therefore equivalent to maximum likelihood under homoskedastic Gaussian observation noise. This is one reason why squared error is the natural benchmark loss for regression; see Bishop (2006) Goodfellow et al. (2016)Deisenroth et al. (2020).

For classification tasks, the model must output class probabilities. In the binary case ($K=2$), the simplest approach passes a single scalar score $z$ through the sigmoid function,

and assigns class 1 whenever $p > 0.5$, equivalently whenever $z > 0$ (Figure Figure 1.7). For $K>2$ classes the natural generalization maps a raw score vector $\bm{z}\in\mathbb{R}^K$ onto the probability simplex via softmax: $\hat{y}_k = e^{z_k}/\sum_j e^{z_j}$. In both cases the misfit between predicted probabilities and true labels is measured by the cross-entropy loss:

where $\hat{y}_k^{(i)}$ is the predicted probability that observation $i$ belongs to class $k$. If the target is encoded as a one-hot vector, exactly one component of $(y_1^{(i)},\dots,y_K^{(i)})$ equals one and all others are zero, so for an observation whose true class is $c$ the loss contribution reduces to $-\log \hat{y}_c^{(i)}$. The model is rewarded for assigning high probability to the correct class and penalized heavily when that probability is near zero.

The origin of cross-entropy is again likelihood theory. In binary classification, with label $y \in \{0,1\}$ and predicted success probability $p$, the Bernoulli log-likelihood is

Negating and averaging this expression gives the binary cross-entropy. The $K$-class formula above is the corresponding negative log-likelihood for a categorical distribution with probabilities generated by sigmoid ($K=2$) or softmax ($K>2$). Cross-entropy is therefore the statistically natural loss whenever the model output is meant to represent class probabilities; see again Bishop (2006) Deisenroth et al. (2020).

Figure Figure 1.8 makes the practical difference visible. If the true label is $y=1$ but the model predicts a very small $p$, then the cross-entropy loss explodes because $-\log p \to \infty$ as $p \downarrow 0$. The same holds symmetrically when $y=0$ and $p \uparrow 1$. Mean squared error does penalize mistakes, but it does so much more mildly near the boundaries. For probabilistic classification, that weaker penalty is usually undesirable because it does not strongly discourage overconfident wrong predictions.

The optimization is performed via gradient descent or one of its stochastic variants, which we discuss in Section Section 1.6.

1.4.1Beyond MSE: Robust and Asymmetric Losses¶

MSE is optimal under Gaussian noise, but real-world economic and financial data often contain outliers and heavy tails that inflate the squared penalty disproportionately. Two classical alternatives are useful in this course and beyond.

1.4.1.0.1Huber loss.¶

Introduced by Huber (1964) in the context of robust location estimation, the Huber loss behaves like MSE near the origin and like $L_1$ in the tails, capping the influence of any single observation:

The threshold $\delta$ controls the transition and is typically chosen to be a few times the noise scale. Huber loss retains the smoothness needed for gradient-based optimization while reducing the weight of extreme residuals, which makes it the default choice for regression problems with suspected outliers.

1.4.1.0.2Quantile (pinball) loss.¶

Koenker & Bassett (1978) proposed the check function

whose minimizer is the conditional $\tau$-quantile of $y$ given $\x$ rather than the conditional mean. Setting $\tau = 0.5$ recovers the median (absolute-error) regression; setting $\tau = 0.05$ or $\tau = 0.95$ targets the lower or upper tail. In financial risk management this is precisely the statistic of interest: $\tau = 0.05$ yields a neural-network estimator of the lower-tail $5\%$-quantile of returns, which corresponds to Value-at-Risk (VaR) at the conventional $5\%$ level, and averaging the pinball loss over many quantiles traces out the full conditional distribution of returns, an approach known as quantile regression or distributional regression.

1.5From Perceptrons to Deep Networks¶

The building block of every neural network is the artificial neuron, first proposed by McCulloch & Pitts (1943). A companion question, how the synaptic weights $w_i$ themselves should adapt with experience, was first addressed by Hebb (1949), whose rule “neurons that fire together, wire together” ($\Delta w_{ij} = \eta\, x_i y_j$) is the conceptual ancestor of all gradient-based learning rules discussed below. Rosenblatt (1958) then introduced the Perceptron, the first trainable binary classifier built on these ideas. A single neuron computes a weighted linear combination of its inputs, adds a bias term, and passes the result through a nonlinear activation function:

where $\w = (w_1, \dots, w_d)^\top$ are the synaptic weights, $w_0$ is the bias, and $g(\cdot)$ is the activation function (Figure Figure 1.9).

Common choices for $g$ include the sigmoid $\sigma(z) = (1+e^{-z})^{-1}$, the hyperbolic tangent $\tanh(z)$, and the rectified linear unit $\mathrm{ReLU}(z) = \max(0,z)$ Nair & Hinton, 2010Glorot et al., 2011. Without a nonlinear activation, any composition of linear layers collapses to a single affine map, a mathematical fact of fundamental importance: $\W_2(\W_1\x + \bb_1) + \bb_2 = \W_2\W_1\x + (\W_2\bb_1 + \bb_2)$.

1.5.0.0.1From a single neuron to a layer.¶

The single-neuron equation $\hat y = g(w_0 + \x^\top \w)$ produces a scalar output. In practice we want vector-valued outputs (and, more importantly, vector-valued intermediate features). A layer of $n$ parallel neurons, each with its own weights $\w^{(j)}$ and bias $w_0^{(j)}$, is a vector-valued generalization

where the nonlinearity $g$ is applied componentwise. Each row of $\W$ is the weight vector of one neuron; stacking $n$ of them gives the matrix $\W$ at once, so the layer evaluates $n$ neurons in a single matrix--vector product.

1.5.0.0.2From one layer to a deep composition.¶

A single hidden layer is already a universal approximator (Cybenko (1989)Hornik et al. (1989); the universal-approximation theorem stated in the next subsection), but its hidden-layer width can grow exponentially in the input dimension to attain a target accuracy. Stacking layers on top of one another reuses earlier features as inputs to later neurons; the resulting compositional representation is dramatically more efficient for many functions of interest Telgarsky, 2016Barron, 1993. A deep feedforward network with $L$ layers is therefore a nested composition of layer maps:

The architecture that implements (1.12) is sketched in Figure Figure 1.10.

A useful geometric intuition, popularized by Chollet (2017), is that each layer of the network performs a nonlinear coordinate transformation, successively “untangling” the manifold on which the data lies. In the input space, the data may be entangled in complex ways (e.g., two classes forming concentric spirals); each hidden layer warps the space so that the data become progressively more linearly separable. By the final hidden layer, a simple linear readout suffices. This perspective, formalized also by Goodfellow et al. (2016), explains why depth is so powerful: each layer adds an additional coordinate transformation, and the composition of many simple transformations can represent very complex mappings with far fewer parameters than a single, wide layer would require.

1.5.0.0.3Other architectures.¶

This course focuses almost entirely on feedforward networks of the form (1.12), because DEQNs and PINNs operate on unstructured state vectors $\x \in \R^d$ for which feedforward maps are the natural choice. For structured inputs other architecture families exist and are used widely elsewhere: convolutional networks LeCun et al., 2015 for image data, graph neural networks for relational data, and Transformers (Section Section 1.11.4) for sequences. We mention them so that readers who encounter these models in the empirical-finance or applied-ML literatures know where they fit; none are required for the methods developed in later chapters.

The universal approximation theorem Cybenko, 1989Hornik et al., 1989 guarantees that even a single hidden layer with sufficiently many neurons can approximate any continuous function on a compact set to arbitrary precision. However, in practice, deep (multi-layer) networks achieve the same accuracy with exponentially fewer parameters than wide (single-layer) ones, which motivates the use of depth; Telgarsky (2016) makes this precise by exhibiting compositional functions that a depth-$L$ network can represent in $\mathrm{poly}(d,L)$ parameters but for which any depth-$(L-1)$ network requires width exponential in $L$. Barron (1993) provides classical dimension-independent approximation rates for Barron-class targets, often stated as squared $L^2$ error of order $\mathcal{O}(1/n_1)$ in the hidden width, whereas tensor-product methods for generic smooth functions scale poorly in the total number of grid nodes. This qualified comparison is the formal version of the “deep learning can beat grids” argument that motivates DEQNs in Chapter Chapter 2.

1.6Training: Loss Functions, Gradient Descent, and Backpropagation¶

Given a loss function $J(\bm{\theta})$, training proceeds by iteratively updating the parameters in the direction of steepest descent:

where $\eta > 0$ is the learning rate. In this introductory chapter $\bm{\theta}$ denotes generic trainable model parameters; in later chapters, when structural parameters and neural-network weights appear together, the script uses $\bm{\theta}$ for the structural parameters and $\rho$ for the network weights. Computing the gradient $\nabla_{\!\bm{\theta}} J$ for a deep network is achieved through backpropagation Rumelhart et al., 1986, an efficient application of the chain rule that propagates error signals from the output layer back to the input layer. Appendix Appendix B collects the matrix-calculus identities and the one-paragraph reverse-mode AD summary used throughout the script.

To see why the chain rule is central, consider a network with a single hidden layer. Let $\z = \W^{(1)}\x + \bb^{(1)}$, $\a = g(\z)$, and $\hat{y} = \w^{(2)\top}\a + b^{(2)}$ with loss $J = \tfrac{1}{2}(\hat{y}-y)^2$. Define the “delta” at the hidden layer:

Then the weight gradient follows immediately:

The key insight of Rumelhart et al. (1986) is that this chain rule application can be organized as a single backward pass through the network, reusing intermediate quantities (the $\bm{\delta}$ vectors) from the forward pass. The resulting algorithm has computational cost proportional to the forward pass; there is no need for finite differences or other expensive gradient approximations.

In practice, evaluating the full gradient over all $m$ training examples is expensive. Stochastic gradient descent (SGD), whose roots go back to the stochastic-approximation scheme of Robbins & Monro (1951), replaces the full sum with a random mini-batch $\mathcal{B} \subset \{1, \dots, m\}$, yielding an unbiased estimate of the empirical-risk gradient at much lower cost per iteration:

With mini-batch sizes of 32--256, SGD achieves both computational efficiency and an implicit regularization effect from gradient noise. Two strands of theoretical work explain why this matters in deep nets specifically: the loss landscape of a deep network is dominated by saddle points rather than isolated bad local minima Dauphin et al., 2014, and SGD’s gradient noise tends to bias training toward flat rather than sharp regions of the loss surface, which often generalize better Keskar et al., 2017. Even on linearly separable data, gradient descent on the logistic loss converges to the maximum-margin solution, an instance of the broader principle that the optimizer itself imposes an implicit bias that contributes to generalization Soudry et al., 2018. For a comprehensive modern review of stochastic optimization for large-scale learning (including convergence rates, adaptive methods, and variance reduction), see Bottou et al. (2018).

1.6.1The Adam Optimizer¶

Modern optimizers such as Adam Kingma & Ba, 2015 adapt the learning rate for each parameter based on running averages of the first and second moments of the gradient:

The bias-corrected first moment $\hat{\bm{m}}_t$ provides momentum (smoothing out gradient noise), while the second moment $\hat{\bm{v}}_t$ provides per-parameter adaptive learning rates (parameters with large gradients receive smaller effective steps). The default hyperparameters $\beta_1=0.9$, $\beta_2=0.999$, $\varepsilon=10^{-8}$ work well across a wide range of problems, including all the economic applications in this course.

1.6.2The Optimizer Family Tree: Momentum, RMSprop, Adam, AdamW¶

Adam did not appear out of thin air; it inherits from a family of refinements to plain SGD whose interactions are worth being explicit about for readers who will tune optimizers in practice. Table Table 1.1 traces the lineage; each row is a one-line modification of the row above it.

The Adam-vs-AdamW distinction is sharper than the one-line table entry suggests, so it is worth writing out both rules side by side. With $\hat m_t$, $\hat v_t$ the bias-corrected first and second moment of the gradient and $\lambda$ the weight-decay rate, Adam-with-$L_2$ (i.e. Adam applied to the loss $J + \tfrac{\lambda}{2}\|\bm\theta\|^2$) updates

so the implicit regularizer is itself rescaled by the adaptive denominator $\sqrt{\hat v_t}+\varepsilon$. AdamW separates the two:

so the weight-decay term shrinks every parameter by the same proportional factor regardless of gradient magnitude. This is why AdamW recovers the textbook intuition “weight decay shrinks weights uniformly” that Adam-with-$L_2$ loses.

Figure Figure 1.11 gives a schematic comparison of the qualitative convergence patterns behind this optimizer family tree.

1.6.3Learning Rate Schedules¶

The choice of learning rate $\eta$ is arguably the single most important hyperparameter in deep learning. Too large, and the optimizer diverges; too small, and convergence is impractically slow. A popular schedule is cosine annealing Loshchilov & Hutter, 2017, which smoothly decays the learning rate according to:

where $T$ is the total number of training iterations. Figure Figure 1.12 compares the three learning-rate strategies used most often in practice.

In practice, decaying schedules such as exponential decay or cosine annealing tend to refine solutions in the later stages of training, once the optimizer has found a good basin of attraction.

1.6.4Backpropagation: The Chain Rule at Scale¶

For a network with $L$ layers, denote the pre-activation at layer $l$ as $\z^{(l)} = \W^{(l)}\a^{(l-1)} + \bb^{(l)}$ and the activation as $\a^{(l)} = g(\z^{(l)})$. If the final layer is linear, set $g'(z)=1$ at $l=L$ and interpret $\a^{(L)}$ as the prediction $\hat y$. The backpropagation algorithm computes $\partial J / \partial \W^{(l)}$ for all layers simultaneously by propagating a “delta” vector backward:

where $\odot$ denotes element-wise multiplication. The parameter gradients are then $\partial J / \partial \W^{(l)} = \bm{\delta}^{(l)} (\a^{(l-1)})^\top$ and $\partial J / \partial \bb^{(l)} = \bm{\delta}^{(l)}$. The computational cost is linear in the number of layers and the total number of parameters, a remarkable efficiency that enables training networks with millions of parameters. Figure Figure 1.13 shows the forward and backward passes side by side.

In modern deep learning frameworks such as TensorFlow and PyTorch, backpropagation is implemented automatically through computational graph tracing (“autograd”). The user only needs to define the forward computation; the framework handles the differentiation. This same automatic differentiation capability is what makes PINNs (Chapter Chapter 7) possible: the PDE residual requires derivatives of the network output with respect to its inputs, which autograd provides exactly and efficiently.

1.7Weight Initialization¶

The initialization of network weights has a profound effect on training dynamics. If weights are initialized too large, activations explode through the layers; if too small, they vanish to zero. In both cases, the gradient signal degrades and training stalls. The key principle is to choose initial weights so that the variance of activations remains approximately constant across layers.

1.7.0.0.1Xavier/Glorot initialization.¶

Glorot & Bengio (2010) derived the following rule for networks with symmetric activations (such as $\tanh$). For a layer with $n_\mathrm{in}$ input neurons and $n_\mathrm{out}$ output neurons, initialize weights as:

This ensures $\mathrm{Var}[a^{(l)}] \approx \mathrm{Var}[a^{(l-1)}]$ under the assumption that activations are in the linear regime of $\tanh$.

1.7.0.0.2He initialization.¶

For ReLU activations, He et al. (2015) showed that the weight variance should be doubled relative to the forward fan-in rule:

The justification is a second-moment-preserving calculation, not a variance one. For a centered symmetric pre-activation $z$,

because $p(z)$ is symmetric about zero, so the negative half of the integrand is killed and the positive half is preserved. Doubling the input weight variance therefore preserves the second moment $\E{(z^{(l)})^2}$ across layers under ReLU. Strictly speaking $\E{\mathrm{ReLU}(z)} > 0$, so the variance (centered second moment) is slightly smaller than the second moment, and the factor of 2 is an approximation rather than an identity; in practice the approximation is excellent and He initialization is the default for ReLU-family networks throughout this course.

1.8Activation Functions in Depth¶

Beyond the three classical choices (sigmoid, tanh, ReLU), several modern activation functions address specific shortcomings. Table Table 1.2 summarizes the options used in this course.

Leaky ReLU and ELU address the dying-neuron issue by providing a small but nonzero gradient for negative inputs. The Swish activation $\mathrm{swish}(z) = z\sigma(z)$ Ramachandran et al., 2017, which is used extensively in the DEQN and IRBC implementations of this course, combines the benefits of ReLU (non-saturating for large $z$) with smoothness at the origin. Its derivative $\mathrm{swish}'(z) = \sigma(z) + z\sigma(z)(1-\sigma(z))$ is smooth everywhere and bounded between approximately -0.1 and 1.1, which can improve optimization stability.

For PDE applications (Chapter Chapter 7), the choice of activation function is particularly important because the PINN loss involves derivatives of the network output. Since $\mathrm{ReLU}''(z) = 0$ almost everywhere, a ReLU network cannot represent second-order PDE residuals faithfully. Smooth activations such as $\tanh$ ($C^\infty$) or Swish are therefore required for PINN applications involving second-order PDEs. Figure Figure 1.14 plots seven representative activations from Table Table 1.2.

1.9Vanishing and Exploding Gradients¶

A central obstacle to training deep networks is that the gradient signal reaching early layers can become either astronomically small or astronomically large as it is back-propagated through many layers. The backward recursion derived above for the “delta” vector is

so the magnitude of $\bm{\delta}^{(l)}$ is governed, roughly, by the product of derivatives $\prod_{k=l}^{L} g'(\z^{(k)})$ and the norms $\|\W^{(k)}\|$. Two symmetric failure modes follow:

• Vanishing gradients. For the sigmoid activation, $|g'(z)| \leq 1/4$; for $\tanh$, $|g'(z)| \leq 1$; and both derivatives approach zero when $|z|$ is large. In the worst sigmoid case each layer shrinks the gradient by a factor close to $1/4$, so after $L=10$ layers the signal at the first layer can be attenuated by roughly $(1/4)^{10}\approx 10^{-6}$. Early layers stop learning.

• Exploding gradients. If $|g'(z)| > 1$ or if $\|\W^{(k)}\|$ is large, the product grows instead of shrinking. Updates become huge, parameters diverge, and the loss “blows up”.

Three ingredients, each already introduced separately, combine to tame these problems:

1. Non-saturating activations. ReLU has $g'(z) = 1$ for $z > 0$, eliminating the $(1/4)^L$ decay; Swish and tanh avoid vanishing when activations remain in a moderate range.

2. Variance-preserving initialization. Xavier/Glorot Glorot & Bengio, 2010 and He He et al., 2015 pick $\mathrm{Var}[W] \propto 1/n_\mathrm{in}$ precisely so that $\mathrm{Var}[\z^{(l)}]$ is constant across layers, keeping activations in the useful range of $g$ (Section Section 1.7).

3. Batch normalization Ioffe & Szegedy, 2015. Re-centering and re-scaling the pre-activations of each mini-batch prevents them from drifting toward the saturated tails of $g$ during training and allows much larger learning rates. Its affine parameters $(\gamma, \beta)$ are learned.

A practical complement is gradient clipping: if $\|\nabla_{\bm{\theta}} J\|$ exceeds a threshold, rescale it to the threshold. This eliminates the most damaging exploding-gradient events at negligible cost and is standard in RNN training; it is occasionally useful in DEQNs and PINNs when the residual magnitudes are highly unbalanced across collocation points.

1.9.1Batch Normalization¶

Among the three mitigations listed above, batch normalization Ioffe & Szegedy, 2015 (BN) deserves a closer look because it is simple to state, surprisingly effective in practice, and has become a default building block in supervised deep networks. At its core BN is the standardization trick familiar from regression, re-centering each variable to mean zero and rescaling it to unit variance, applied separately to every layer’s pre-activations and recomputed on every mini-batch of training data.

Let $z_1,\dots,z_B$ denote the pre-activations at one neuron over the $B$ examples in a mini-batch $\mathcal B$. Batch normalization replaces them by

where $\varepsilon$ is a small constant for numerical stability and $(\gamma,\beta)$ are learnable scalar parameters specific to that neuron. The transformed activation $y_i$ is what the next layer sees. In the standard recipe BN is inserted between the linear map $\W^{(\ell)}\bm a^{(\ell-1)}+\bm b^{(\ell)}$ and the elementwise nonlinearity $g(\cdot)$.

1.9.1.0.1Why standardization, layer by layer.¶

Without BN, the input distribution to a hidden layer $\ell$ depends on every weight in layers $1,\dots,\ell-1$. As earlier weights update during gradient descent, the distribution faced by layer $\ell$ drifts from one optimization step to the next: each layer therefore chases a moving target, a phenomenon Ioffe & Szegedy (2015) called internal covariate shift. BN pins the input distribution of every layer to mean zero and unit variance at every step (Figure Figure 1.15). Gradients become better conditioned, and substantially larger learning rates become safe.

1.9.1.0.2The role of the affine parameters.¶

At first glance the learnable shift and scale $(\gamma,\beta)$ seem to undo the normalization that BN just imposed. This is exactly the point. If a layer happens to prefer non-standard inputs, for example a tanh layer that needs slightly negative pre-activations to operate in its linear regime, the network is free to recover them via $(\gamma,\beta)$. BN therefore never reduces the network’s representational capacity; it merely shifts to a parameterization in which the optimization trajectory is easier to follow.

1.9.1.0.3Why higher learning rates work.¶

A precise Lipschitz bound depends on the operator norms of the surrounding weight matrices, the activation derivatives, and the learned affine scale $\gamma$. The useful intuition is that BN reduces sensitivity to shifts and rescalings of intermediate activations, making the local optimization problem better conditioned and allowing step sizes that would otherwise cause divergence. Santurkar et al. (2018) argue that loss-landscape smoothing, rather than the original “internal covariate shift” interpretation, better explains why BN helps optimization; the two views are complementary, but the smoothing perspective is the more directly testable one.

1.9.1.0.4At inference time.¶

During training, BN uses the current mini-batch to compute $\mu_{\mathcal B},\sigma^2_{\mathcal B}$. At inference the mini-batch may be a single example, in which case those statistics would be ill-defined. Implementations therefore maintain a running average of $\mu$ and $\sigma^2$ across training mini-batches and use these fixed estimates at test time, so the network’s output is deterministic at deployment.

1.9.2Normalization Variants Beyond BatchNorm¶

Batch normalization is the original normalization trick, but for several common deep-learning settings, small mini-batches, recurrent / sequence models, generative models, transformers, it is not the right one. All variants share the form $\hat z = (z-\mu)/\sigma$ followed by a learned affine $\gamma\hat z + \beta$; they differ only in which axes the statistics $(\mu,\sigma)$ are pooled over:

• LayerNorm Ba et al., 2016: pool across all features of a single example. Decouples training from batch size and is the de-facto choice for RNNs and transformers; it can also be useful in small-batch residual methods, though PINNs more commonly rely on careful input/output scaling and smooth activations.

• GroupNorm Wu & He, 2018: pool over a group of channels of a single example. Interpolates between LayerNorm ($G\!=\!1$) and InstanceNorm ($G$ = channels); the default in object detection and small-batch CNNs.

• WeightNorm Salimans & Kingma, 2016: reparameterize $\bm w = (g/\|\bm v\|)\,\bm v$ so the activation statistics never enter the gradient. Avoids any batch-size dependence at the cost of slightly less representation power.

For the methods in this script, the practical guidance is: use BatchNorm for large supervised datasets (Chapter Chapter 1), LayerNorm for recurrent or attention-based architectures and as an option when the effective batch size is small, and skip normalization entirely for small DEQN or PINN MLPs when input/output scaling plus Adam already condition the problem well.

1.10Generalization: Overfitting, Regularization, and Double Descent¶

1.10.1Train / Validation / Test Split¶

Before discussing overfitting formally, it is essential to fix the experimental protocol that every supervised-learning study should follow. The available data are partitioned into three disjoint subsets:

• Training set (typically $\sim 70\%$): used to fit the model parameters $\bm{\theta}$ by minimizing the loss.

• Validation set (typically $\sim 20\%$): used for model selection, choosing hyperparameters, comparing architectures, deciding when to stop training. The model’s performance on this set guides these decisions but its parameters are never trained on it.

• Test set (typically $\sim 10\%$): touched once, at the very end, to report an unbiased estimate of generalization performance.

The key discipline is that no decision about the model (not hyperparameter tuning, not architecture, not early-stopping patience) may be informed by the test set. Using the test set multiple times turns it into an implicit validation set and invalidates its role as a measure of out-of-sample error. For small datasets, $k$-fold cross-validation replaces the fixed train/validation split: the training data are partitioned into $k$ equal folds; for each fold, the model is trained on the other $k-1$ folds and evaluated on the held-out fold; the $k$ resulting validation scores are averaged. Common choices are $k = 5$ or $k = 10$; the test set is always held separately. In DEQNs and PINNs (Chapters Chapter 2 and Chapter 7), training and validation points are drawn from the same state distribution, and “generalization” is measured against an independently simulated test trajectory rather than a held-out labeled set.

A model that memorizes the training data but fails on unseen examples is said to overfit. To understand overfitting precisely, consider the following thought experiment. The decomposition below is the classical bias/variance analysis of Geman et al. (1992), which provided the canonical framework for thinking about generalization in neural networks long before modern overparameterized regimes were studied. Suppose we draw many independent training sets $\mathcal{D}$, each of size $n$, from the same data-generating process $y = f(\x) + \varepsilon$, where $\varepsilon$ is zero-mean noise with variance $\sigma^2$. On each training set we fit our model, obtaining a predictor $\hat{f}_{\mathcal{D}}$. Conditioning on a fixed test input $\x_0$ and averaging over both the training set and the new test noise $\varepsilon_0$, the squared prediction error decomposes into exactly three terms:

Each term captures a distinct source of error:

• Bias$^2$ measures the systematic error: how far the average prediction $\mathbb{E}_{\mathcal{D}}[\hat{f}_{\mathcal{D}}(\x_0)]$ is from the true function $f(\x_0)$. A model that is too simple (e.g., a linear function fit to a nonlinear target) will have high bias regardless of how much data it sees.

• Variance measures the sensitivity of the prediction to the particular training set drawn. A highly flexible model (e.g., a large neural network) may fit each training set well, but the predictions can differ wildly across draws, which is overfitting.

• Irreducible noise $\sigma^2$ is the error that no model can eliminate, because it stems from randomness in the data-generating process itself.

In classical statistics, there is a fundamental trade-off: reducing bias requires more flexible models, which increases variance. However, modern deep networks challenge this picture. Zhang et al. (2017) showed empirically that standard architectures can perfectly fit randomly labeled data, implying a VC-style capacity far beyond what classical bounds predict, yet still generalize well on real data. Belkin et al. (2019) subsequently demonstrated that with sufficient overparameterization, models exhibit a double descent phenomenon: test error first increases as the model becomes more complex (classical regime), but then decreases again as the number of parameters greatly exceeds the number of data points (interpolation regime). Nakkiran et al. (2020) showed that this phenomenon extends to deep networks and occurs not only as a function of model size, but also as a function of training time (“epoch-wise double descent”) and dataset size.

The key techniques for preventing overfitting in neural networks are:

1. Early stopping Prechelt, 1998: monitor validation loss and stop training when it begins to rise.

2. Weight decay ($L_2$ regularization) Krogh & Hertz, 1991: add $\frac{\lambda}{2}\|\bm{\theta}\|^2$ to the loss.

3. Dropout Srivastava et al., 2014: randomly drop a fraction $p$ of activations at every training step. Two implementation conventions exist. The original convention drops units during training and multiplies the outgoing weights or activations by the keep probability $1-p$ at test time, so that the expected activation matches the training-time expectation. The now-standard inverted-dropout convention divides the retained activations by $1-p$ during training, so no rescaling is needed at test time. Either way, the mechanism is equivalent to training, on each mini-batch, a different sub-network drawn from an exponentially large ensemble that shares weights; the final network approximates the ensemble average. Typical values are $p=0.5$ for hidden layers and $p=0.1$--0.2 for inputs. Dropout is less commonly used in DEQN and PINN applications, where the loss is already noisy (stochastic collocation) and regularization is often supplied implicitly by the state-space sampling scheme.

4. Data augmentation: synthetically enlarge the training set via transformations.

5. Batch normalization Ioffe & Szegedy, 2015: normalize activations within each mini-batch to stabilize training; its mini-batch statistics also act as a mild regularizer.

Figure Figure 1.16 illustrates why classical bias--variance intuition breaks down for modern deep networks. In the classical regime ($p < n$), increasing model capacity beyond a point leads to overfitting. At the interpolation threshold ($p \approx n$), the model has just enough parameters to perfectly fit the training data, and the resulting solution can be extremely sensitive to noise. In the modern regime ($p \gg n$), test error often decreases again because optimization and architecture bias select comparatively regular interpolating solutions rather than arbitrary ones Belkin et al., 2019.

This phenomenon has been documented across many architectures and datasets by Nakkiran et al. (2020), who showed that it persists even when controlling for effective model complexity. The implications for computational economics are substantial but should not be overstated. In DEQN and PINN applications, the practitioner controls both the network size (number of parameters $p$) and the amount of training data (number of collocation points $n$), and those collocation points are often resampled rather than fixed once and for all. Overparameterized networks can therefore be useful and sometimes necessary, but their credibility must be checked by independent residual diagnostics, simulated trajectories, and benchmark comparisons rather than by parameter counting alone.

The double descent phenomenon can be explored interactively in the companion notebook 03_Double_Descent.ipynb, which reproduces the spirit of the double-descent curve in a small kernel-regression / random-Fourier-features setting (Belkin et al. (2019)); see also Nakkiran et al. (2020) for the deep-network / CIFAR-10 version of the experiment, which the notebook does not attempt to replicate at scale.

1.10.2A Pointer to the Theory: Neural Tangent Kernel (NTK) and Benign Overfitting¶

Why does an over-parameterized network with $p \gg n$ generalize instead of merely memorizing? Two complementary lines of theory have emerged.

1.10.2.0.1Neural Tangent Kernel (NTK).¶

In the limit of infinite width and a particular initialization scaling, gradient-descent training of a deep network is described by kernel gradient descent with a fixed, deterministic kernel, the Neural Tangent Kernel of Jacot et al. (2018). In that lazy-training limit the network’s function evolves almost linearly around its initialization, which explains why first-order optimization can be well behaved for very wide networks even though the finite-parameter objective is non-convex.

1.10.2.0.2Benign overfitting.¶

In linear and kernel settings, gradient methods often select a minimum-norm interpolating solution, which can behave much like a ridge-regularized least-squares estimator and inherit good generalization properties. Bartlett et al. (2020) make this precise for linear regression, showing that interpolation can be benign provided the spectrum of the input covariance has heavy enough tails. Subsequent work has extended both stories beyond the simplest settings; for the practitioner the takeaway is that the NTK regime helps explain why training succeeds, while benign-overfitting theory explains why interpolation need not imply poor test error under additional assumptions.

These two threads are not the final word: finite-width deviations from the NTK matter for feature learning, and benign overfitting requires conditions on the data covariance. They nevertheless help explain why the rest of this script can use networks substantially wider than a classical degrees-of-freedom calculation would recommend, provided the numerical residuals are validated out of sample.

1.11Sequence Models: RNNs, LSTMs, and Attention¶

The chapters that follow rely almost entirely on feedforward networks, but many economic and financial datasets are intrinsically sequential. Before closing this introduction, it is therefore useful to briefly survey the main architectures for sequence data. We represent a sequence as tokens $\x_1, \dots, \x_n$, where each token is a vector. The word “token” is borrowed from natural-language processing, where a token is typically a word or sub-word piece; in the economic and financial context we use throughout this course a token is simply one element of the sequence, for example a scalar return, a vector of macroeconomic variables at one quarter, or a price-volume pair at one tick. The algorithms below are agnostic to this choice; they only need the sequence elements to be real-valued vectors of a fixed dimension.

1.11.1Recurrent Neural Networks¶

The traditional approach to sequence data is the Recurrent Neural Network (RNN) Elman, 1990. Unlike an MLP, which maps an input to an output in a single forward pass, an RNN maintains an internal hidden state $\h_t$ that acts as a memory of past information. At each time step $t$, the network updates this state using the current input $\x_t$ and the previous state $\h_{t-1}$:

where $\sigma$ is an activation function. Concretely: for a scalar time series $\x_t$ is a scalar (e.g. log-return at date $t$), $\h_t \in \R^d$ is a $d$-dimensional hidden vector summarizing everything the network has seen so far, and $\Wh,\Wx,\bb$ are learnable parameters. The same update is applied at every time step, so this recursive structure lets the network process sequences of arbitrary length with a fixed parameter budget. Figure Figure 1.17 shows the resulting unrolled computation graph.

1.11.1.0.1Training: Backpropagation Through Time (BPTT).¶

Because the unrolled RNN is a feedforward graph of depth $T$ with shared weights, one can apply ordinary backpropagation and then sum the weight-gradients across time. Concretely, let $\mathcal{L}_T = \sum_{t=1}^{T}\ell(\hat{\y}_t, \y_t)$ denote the total loss. For the recurrence above, the forward single-step Jacobian is

With column gradients, the backward pass multiplies by $J_t^\top$. Differentiating with respect to an early hidden state $\h_k$ therefore yields the schematic product

so the gradient picks up $T-k$ products of matrices containing the same recurrent weight matrix $\Wh$. The relevant singular values or spectral radii of these factors determine the asymptotics. If they are mostly below one, the gradient vanishes exponentially in $T-k$ and the network cannot learn dependencies that span many steps; if they are mostly above one, it explodes, producing NaNs during training. This is the vanishing/exploding gradient problem, originally analyzed by Hochreiter (1991) and developed formally in Bengio et al. (1994) Hochreiter et al. (2001), and revisited with a modern optimization-theoretic view by Pascanu et al. (2013).

Three practical remedies partially alleviate the pathology without changing the architecture. Gradient clipping Pascanu et al., 2013 rescales the parameter-gradient whenever its norm exceeds a threshold; it eliminates the worst exploding events at negligible cost and is now a standard training default for any recurrent model. Orthogonal or identity-like initialization of $\Wh$ places its singular values near 1 so that the Jacobian product preserves norms at the start of training. Truncated BPTT unrolls only $K$ steps back at each update, capping both the memory footprint and the effective gradient horizon at $K$. These help but do not cure the problem: the structural fix is to change the recurrence itself, which is the move made by gated cells.

1.11.2Long Short-Term Memory (LSTM) and GRUs¶

Before writing equations, it helps to keep a plain-language picture in mind. A vanilla RNN asks one hidden state $\h_t$ to do three jobs at once: retain useful old information, absorb new information, and expose the relevant part to the next layer. This is a fragile design. The LSTM separates these tasks. It keeps a dedicated memory lane $\bm{C}_t$ flowing through time and at each date makes three soft decisions: what to keep, what to write, and what to reveal. That is the intuition behind the gate equations below.

The LSTM cell Hochreiter & Schmidhuber, 1997 replaces the single-state recurrence $\h_t = \sigma(\Wh\h_{t-1} + \Wx\x_t)$ by a pair of states: a cell state $\bm{C}_t$ that flows along the top of the cell with additive updates, and a hidden state $\h_t$ that is read off it. Three learned sigmoid gates, each depending on the concatenation $[\h_{t-1}, \x_t]$, control what information flows through:

Each of $\bm{f}_t, \bm{i}_t, \bm{o}_t \in (0,1)^{d}$ acts as a soft switch applied element-wise. The crucial structural change is in equation (1.34): the cell state is additively corrected rather than multiplicatively overwritten. Along the direct memory path, differentiating $\bm{C}_t$ with respect to $\bm{C}_{t-1}$ contributes $\mathrm{diag}(\bm{f}_t)$ in place of a full recurrent matrix product. The full derivative also contains indirect terms because the gates depend on $\h_{t-1}$ and hence on earlier cell states, but the direct path is the constant-error-carousel intuition: when the cell judges information worth keeping, it can open the forget gate ($\bm{f}_t \approx \bm{1}$) and allow gradients to flow through as if the sequence were shorter. Figure Figure 1.18 sketches the resulting cell.

The Gated Recurrent Unit (GRU) of Cho et al. (2014) is a lighter sibling that merges the forget and input gates into a single update gate $\bm{z}_t$ and drops the separate cell state in favor of the hidden state itself:

A GRU uses roughly $25\%$ fewer parameters than an LSTM of the same hidden size and performs comparably on many sequence tasks Chung et al., 2014, at the cost of a slightly less expressive memory channel.

LSTMs remain particularly effective for economic time-series tasks where capturing specialized temporal patterns is more important than massive scalability. For example, Holt et al. (2024) use LSTMs to detect Edgeworth cycles in retail gasoline-price data. These cycles are asymmetric, high-frequency price movements that are difficult to identify with traditional spectral analysis or simple rule-based methods. In this setting the LSTM architecture excels at identifying the characteristic sawtooth patterns (sudden price jumps followed by slow decays) across thousands of retail stations, providing a robust tool for antitrust analysis and competition policy.

1.11.3Limits of Recurrent Models¶

Even with gated cells, recurrent architectures retain two fundamental limitations that no amount of tuning can remove.

1. Inherently sequential. Step $t$ cannot begin until step $t-1$ has finished, so the full computation takes $T$ serial wavefronts regardless of how many cores or tensor units are available. On a modern GPU that can evaluate thousands of MLP rows in parallel, this forced serialization makes RNN training slow and wall-clock expensive. Training a contemporary large language model on a trillion-token corpus with a plain LSTM would take years of wall-clock time rather than weeks.

2. Path-length-dependent decay. Information from position 1 must travel through all intermediate hidden states to influence position $T$, each hop applying a matrix and a nonlinearity. Gating mitigates but does not eliminate this decay, and empirically LSTM performance saturates on sequences well below the theoretical limit implied by the cell’s capacity.

Both problems have the same structural cause: the recurrence forces a path of length $T$ between the two extreme positions of the sequence. The remedy is to drop recurrence entirely and let every position read every other position directly, in parallel. This is the Transformer.

1.11.4The Transformer Architecture¶

The Transformer architecture Vaswani et al., 2017 replaced recurrence entirely with self-attention. This allows the model to weigh the importance of all tokens in a sequence simultaneously, enabling massive parallelization and better capturing global dependencies.^[1]

For a first pass, four steps are enough. Add position information so the model knows order; project each token into a query, key, and value; compare each query to all keys; then take a weighted average of the values. Multi-heads, LayerNorm, residual connections, and pointwise MLPs are refinements of this core idea, not a different idea.

1.11.4.1The Self-Attention Mechanism¶

1.11.4.1.1Intuition: search, then retrieval.¶

Before writing the mechanism formally it is useful to see it as a soft library lookup. Picture a small library with $n$ shelves, one per position in the sequence. Each shelf $i$ carries three vectors: a query $q_i$ (“what am I looking for?”), a key $k_i$ (“what is printed on my label?”), and a value $v_i$ (“what do I actually contain?”). All three are linear projections of the same input token $\x_i$, and the projection matrices $\W_Q,\W_K,\W_V$ are learned during training; the librarian (queries and keys) and the books (values) are co-designed for whatever task the training objective encodes.

To produce the updated representation $\bm{o}_i$ at shelf $i$, the following four steps happen.

1. Score. Shelf $i$’s query $q_i$ is compared against every shelf’s key $k_j$; the dot product $q_i^{\!\top} k_j$ is a similarity score, high when shelf $j$’s label matches what shelf $i$ is looking for.

2. Normalize. The $n$ scores are divided by $\sqrt{d_k}$ (to keep them in a numerically sane range) and pushed through a softmax, producing probability weights $\alpha_{ij}\in[0,1]$ with $\sum_j \alpha_{ij}=1$.

3. Retrieve. The weights are applied to the values: $\bm{o}_i = \sum_j \alpha_{ij}\, v_j$ is a weighted average of the $n$ shelf contents.

4. Repeat. Steps 1--3 happen in parallel for every shelf $i$, producing the whole output sequence $\bm{o}_1,\dots,\bm{o}_n$ in a single layer.

Shelves whose label matched the query contribute most to the output; shelves whose label was off-topic are almost ignored.

1.11.4.1.2Why this is useful: a worked example.¶

Consider the sentence “The cat sat on the mat. It purred.” For the model to process “it” properly, it must first decide what “it” refers to, the cat or the mat. Self-attention performs exactly this disambiguation: the query vector at the “it” position probes the key vectors at every earlier position, and the softmax converts the raw similarity scores into probability weights that concentrate most of the mass on the correct antecedent. Figure Figure 1.19 below illustrates the resulting pattern: the bulk of the weight lands on “cat”, and the updated representation at the “it” position is formed as a weighted average of the values, driven mostly by “cat”. The same mechanism, run in parallel for every position, produces all of $\bm{o}_1,\dots,\bm{o}_n$ in a single layer.

1.11.4.1.3A small concrete example.¶

Suppose we have only three shelves and two-dimensional $q$’s and $k$’s. Take the query at shelf 3 to be $q_3 = (1,0)$ and the three keys $k_1 = (0.9,\,0.1),\ k_2 = (0.1,\,0.8),\ k_3 = (1,\,0).$ The raw similarity scores are $q_3^{\!\top}k_1 = 0.9$, $q_3^{\!\top}k_2 = 0.1$, $q_3^{\!\top}k_3 = 1.0$. Ignoring the $1/\sqrt{d_k}$ scaling to keep the arithmetic clean, the softmax weights come out to roughly $(0.405,\,0.182,\,0.448)$. Shelf 3’s output $\bm{o}_3$ is therefore a blend of the three values in those proportions: most mass on shelf 3 itself and on shelf 1 (the closest match in label space); shelf 2, whose label $k_2$ points in an orthogonal direction, contributes about $18\%$. The softmax is “soft” precisely in this sense, a smoothed nearest-neighbor retrieval rather than a hard argmax over the most similar shelf.

The “self” in *self-*attention says that every shelf plays both roles simultaneously: every shelf’s query probes every shelf’s key, and every shelf receives an updated representation as output. A single attention layer therefore pairs all $n$ positions with all $n$ positions in parallel. Contrast this with an RNN or LSTM: to let position $t$ peek at information from position 1, the signal must be shuttled through every intermediate position via the hidden state, with each hop applying its own weight matrix and nonlinearity, so long-range information is either blurred or lost entirely. Attention short-circuits that chain: position $t$ reads position 1 directly, with no intermediate hops and no path-length-dependent attenuation. This is the architectural source of the Transformer’s advantage on long sequences, and the reason why attention-based models quickly displaced recurrent ones for language, code, and (increasingly) time-series forecasting.

Given a sequence of input vectors, stack the tokens as rows of $\X \in \R^{n\times d}$. The attention mechanism computes three projections for each token: a Query ($Q$), a Key ($K$), and a Value ($V$), using learned weight matrices $\W_Q, \W_K, \W_V$:

The output of the attention layer is a weighted sum of the values, where the weights are determined by the compatibility (dot product) of the queries with the keys:

The scaling factor $\sqrt{d_k}$ (the dimensionality of the keys) prevents the dot products from growing too large in magnitude, which would otherwise make the softmax unstable.

1.11.4.1.4An econometric lens.¶

The attention layer is a data-dependent, learnable kernel smoother. Compare it to the Nadaraya--Watson estimator $\hat f(x) = \sum_i w_i(x)\,y_i$ with kernel-based weights $w_i(x) \propto k(x, x_i)$: attention has exactly this form, but the similarity $k(\cdot,\cdot)$ is the parametric bilinear form $(q,k)\mapsto q^{\!\top}k/\sqrt{d_k}$ and both $q$ and $k$ are themselves learned projections of the input. From this vantage point the Transformer’s “magic” is less mysterious: it is a nonparametric smoother whose kernel the optimizer tunes to whatever task the training objective encodes. Self-attention further recovers the classical recurrence-free property that every pair of positions interacts in a single parallel layer, with no signal decay along the sequence.