B Matrix Calculus and Automatic Differentiation

This appendix collects the few matrix-calculus identities used in the main text and gives a one-page summary of how automatic differentiation realizes them computationally.

B.1Matrix calculus identities¶

For $\bm A \in \R^{m\times n}$, $\bm x \in \R^n$, $\bm y = \bm A\bm x$:

For a quadratic form $f(\bm x) = \bm x^\top \bm Q \bm x$ with symmetric $\bm Q$, $\nabla f(\bm x) = 2\bm Q\bm x$ and $\nabla^2 f = 2\bm Q$. For a deep network $\bm{\hat y} = g_L(\bm W_L\, g_{L-1}(\cdots g_1(\bm W_1 \bm x + \bm b_1)\cdots))$, the chain rule gives

with $\bm \delta^{(l)} = \bigl(\bm W^{(l+1)\top}\bm \delta^{(l+1)}\bigr) \odot g'(\bm z^{(l)})$ (Section 1.6).

B.2Reverse-mode AD in one paragraph¶

Reverse-mode AD evaluates the function once forward, caches every intermediate value, and then traverses the computation graph backwards, accumulating the local Jacobian--vector products in linear time in the number of operations. For a scalar loss, one reverse sweep returns the gradient with respect to all parameters at a small constant multiple of the forward cost. The computation still scales with the size of the graph, but it does not require one separate derivative pass per parameter; this is what makes million-parameter networks trainable. See Baydin et al. (2018) and Margossian (2019) for complete treatments.

B.3Higher-order AD¶

PINN losses involve second derivatives ($V_{SS}$ in Black--Scholes, Laplacians in Poisson equations, and second-order terms in diffusion HJBs). Higher-order derivatives are obtained by composing reverse-mode AD with itself, torch.autograd.grad of grad in PyTorch, jax.grad of jax.grad in JAX. Activation regularity matters: the network must be at least $C^k$ for the strong $k$-th-order residual to be well-defined.