Twenty books, one notebook.
Reading notes for every title on the shelf: what each book covers section by section, the results worth memorizing, where it connects to ML, and questions to test yourself. Tick off sections as you go; progress is saved in this browser.
- Level
- Upper undergraduate, proof-based
- Prerequisites
- Matrix arithmetic, comfort reading proofs
- Style
- Terse, rigorous, Soviet school; few computations, many structural results
Shilov builds linear algebra from determinants up to the Jordan canonical form and the spectral theory of quadratic forms. The book treats spaces and operators abstractly but always returns to matrices, which makes it a strong second course after a computational first one.
Section notes
Determinants
Defined through permutations rather than cofactor recursion, so every property (multilinearity, alternation, \(\det AB=\det A\det B\)) falls out of one formula:
\[\det A=\sum_{\sigma\in S_n}\operatorname{sgn}(\sigma)\prod_{i=1}^n a_{i\,\sigma(i)}\]
Cofactor expansion and Cramer's rule follow. Read this carefully: the permutation viewpoint returns in exterior algebra and volume forms.
Linear spaces
Axioms, linear dependence, bases, dimension, subspaces, sums and direct sums. Key identity:
\[\dim(U+W)=\dim U+\dim W-\dim(U\cap W)\]
Isomorphism theorem: every \(n\)-dimensional space over \(K\) is isomorphic to \(K^n\), so coordinates are always available but never canonical.
Systems of linear equations
Rank of a matrix as the maximal number of independent rows (equals columns). The Kronecker–Capelli theorem: \(Ax=b\) is solvable iff \(\operatorname{rank}A=\operatorname{rank}[A\mid b]\). The homogeneous solution space has dimension \(n-r\), and the general solution is a particular solution plus that space.
Linear functions and operators
Linear forms, linear operators and their matrices in a basis, kernel and image, rank–nullity \(\dim\ker A+\dim\operatorname{im}A=n\). Invariant subspaces, eigenvectors, and the characteristic polynomial \(\det(A-\lambda I)\). Operators with \(n\) distinct eigenvalues are diagonalizable.
Coordinate transformations
Under a change of basis with matrix \(C\), an operator's matrix becomes \(A'=C^{-1}AC\) and a bilinear form's matrix becomes \(B'=C^{\mathsf T}BC\). Knowing which law applies is the whole difference between operators and forms. Trace, determinant and the characteristic polynomial are similarity invariants.
Canonical form of an operator
The space splits into generalized eigenspaces (root subspaces); on each, \(A-\lambda I\) is nilpotent and decomposes into cyclic blocks. Result: the Jordan normal form. Shilov also develops \(\lambda\)-matrices and elementary divisors, the algebraic route to the same answer, plus the minimal polynomial (Jordan form is diagonal iff the minimal polynomial has simple roots).
Bilinear and quadratic forms
Reduction of a quadratic form to a sum of squares (Lagrange's method, Jacobi's method via leading minors). Sylvester's law of inertia: the counts of positive, negative and zero coefficients are invariants. Sylvester's criterion: a symmetric \(A\) is positive definite iff all leading principal minors are positive.
Euclidean and unitary spaces
Inner products, Cauchy–Schwarz, Gram–Schmidt, orthogonal complements, and the Gram determinant as squared volume. Adjoint operators. The spectral theorem: a self-adjoint operator has real eigenvalues and an orthonormal eigenbasis. Unitary and normal operators. Extremal properties of eigenvalues via the Rayleigh quotient and the Courant minimax principle:
\[\lambda_k=\min_{\dim V=k}\ \max_{x\in V,\,x\ne0}\frac{\langle Ax,x\rangle}{\langle x,x\rangle}\]
Simultaneous diagonalization of two quadratic forms when one is positive definite.
Algebras, representations, categories
Closing chapters on finite-dimensional algebras, their representations, and a short introduction to categories of vector spaces. Optional on a first pass; read once you start group or representation theory.
Key results to memorize
- Rank–nullity and Kronecker–Capelli.
- Operator matrices transform by similarity, form matrices by congruence.
- Jordan form exists over \(\mathbb C\); diagonalizable iff minimal polynomial is squarefree.
- Sylvester's inertia law and criterion.
- Spectral theorem and minimax characterization of eigenvalues.
ML connection. PCA is the spectral theorem applied to a covariance matrix, and the top principal direction maximizes the Rayleigh quotient. Positive definiteness (Sylvester) is how you check a Hessian at a minimum. Shilov does not cover SVD as its own topic; pair it with a short SVD treatment.
Check yourself
- Why can't a real rotation matrix of the plane (not \(\pm I\)) be diagonalized over \(\mathbb R\)?
Answer
Its characteristic polynomial \(\lambda^2-2\cos\theta\,\lambda+1\) has no real roots when \(\sin\theta\ne0\).
- Give a matrix whose characteristic and minimal polynomials differ.
Answer
\(I_2\): characteristic \((\lambda-1)^2\), minimal \(\lambda-1\).
- Is \(\begin{pmatrix}2&1\\1&1\end{pmatrix}\) positive definite?
Answer
Yes: leading minors are \(2>0\) and \(1>0\).
- Level
- Undergraduate, geometric
- Prerequisites
- Analytic geometry, basic linear algebra
- Style
- Index notation throughout; short with many worked problems
A compact course that rebuilds linear algebra in tensor language. Its real lesson is the transformation law: a tensor is defined by how its components change when the basis changes. The cover's ellipsoid is the principal-axes problem for a symmetric tensor, which the book solves.
Section notes
Vectors, bases, summation convention
Einstein convention: a repeated upper and lower index is summed, \(x=x^ie_i\). The reciprocal (dual) basis is defined by
\[e^i\cdot e_j=\delta^i_j\]
Components with upper indices are contravariant, with lower indices covariant. In an orthonormal basis the distinction disappears, which is why many courses never mention it.
Change of basis
If \(e_{i'}=A^i_{i'}e_i\), then contravariant components transform with the inverse matrix and covariant ones with the direct matrix:
\[x^{i'}=A^{i'}_i x^i,\qquad x_{i'}=A^i_{i'}x_i,\qquad A^{i'}_iA^i_{j'}=\delta^{i'}_{j'}\]
This bookkeeping is the core skill of the book; drill it until it is automatic.
The metric tensor
\(g_{ij}=e_i\cdot e_j\) encodes lengths and angles in any basis: \(x\cdot y=g_{ij}x^iy^j\). Its inverse \(g^{ij}\) raises indices, \(x^i=g^{ij}x_j\), and \(g_{ij}\) lowers them. \(\det(g_{ij})\) is the squared volume of the basis parallelepiped.
Linear transformations as rank-2 tensors
An operator has components \(a^i_j\) and transforms as a mixed tensor of type (1,1). Eigenvalues come from \(\det(a^i_j-\lambda\delta^i_j)=0\); the trace \(a^i_i\) and determinant are invariants because they are full contractions.
Symmetric tensors and quadrics
A symmetric bilinear form \(a_{ij}\) can be brought to principal axes by an orthogonal change of basis. Geometric payoff: classify second-order curves and surfaces (ellipsoids, hyperboloids, paraboloids) by the signs of the eigenvalues.
General tensors and tensor algebra
Tensors of type \((p,q)\), addition, tensor product, contraction, symmetrization and alternation. The Levi-Civita symbol and the cross product \( (a\times b)_i=\varepsilon_{ijk}a^jb^k \) (in an orthonormal basis), and the triple product as a determinant.
Key results to memorize
- Contravariant components use the inverse of the basis-change matrix.
- Full contractions are invariants (trace, \(g_{ij}x^ix^j\)).
- Symmetric tensors diagonalize orthogonally; signs of eigenvalues classify quadrics.
ML connection. torch.einsum("ij,jk->ik", A, B) is exactly Einstein notation, and reading this book makes einsum fluent. Note that a deep-learning "tensor" is just a multidimensional array; the transformation law here is what makes the physics and differential-geometry sense of the word. Natural-gradient methods use a metric \(g_{ij}\) (the Fisher information) to raise the gradient's index.
Check yourself
- In a basis \(e_1=(1,0), e_2=(1,1)\), find \(g_{ij}\).
Answer
\(g_{11}=1, g_{12}=g_{21}=1, g_{22}=2\).
- Why is \(a^i_i\) basis-independent but \(a^i_j\) for fixed \(i,j\) is not?
Answer
The contraction absorbs one direct and one inverse transformation factor, which cancel via \(A^{i'}_iA^i_{j'}=\delta\).
- Level
- First course in abstract algebra
- Prerequisites
- None formally; patience with definitions
- Style
- Slow and careful, lots of motivation and examples
Landin introduces groups, rings and fields with unusual care for beginners, building every concept from sets and mappings. It is the gentler on-ramp; Clark is the faster, denser follow-up.
Section notes
Sets, relations, mappings
Injective, surjective, bijective maps, composition, inverses. Equivalence relations and the partition into equivalence classes: the single most reused idea in the book (cosets, quotient rings, \(\mathbb Z/n\mathbb Z\)).
Groups
Axioms, subgroups, cyclic groups, permutation groups. Cosets and Lagrange's theorem \(|G|=[G:H]\,|H|\). Normal subgroups, quotient groups, homomorphisms and the first isomorphism theorem:
\[G/\ker\varphi\;\cong\;\operatorname{im}\varphi\]
Rings and integral domains
Rings, subrings, ideals, quotient rings, ring homomorphisms. Integral domains (no zero divisors) and the construction of the field of fractions, which is how \(\mathbb Q\) is built from \(\mathbb Z\).
Polynomial rings and factorization
Division algorithm and Euclidean algorithm in \(K[x]\), greatest common divisors, irreducible polynomials, unique factorization. \(K[x]/(p)\) is a field exactly when \(p\) is irreducible.
Fields and extensions
Prime fields, algebraic elements, minimal polynomials, degree of an extension and the tower law \([L:K]=[L:M][M:K]\). Applications to ruler-and-compass constructions: constructible numbers live in extensions of degree \(2^k\), which rules out trisecting an arbitrary angle and doubling the cube.
Toward Galois theory
Automorphisms of field extensions and the correspondence between subgroups and intermediate fields, presented as an introduction. Clark's book completes this story.
Key results to memorize
- Lagrange's theorem and the first isomorphism theorem.
- \(K[x]/(p)\) is a field iff \(p\) is irreducible.
- Tower law, and why \(\sqrt[3]{2}\) is not constructible (degree 3 is not a power of 2).
ML connection. Group theory is the language of equivariant networks: a CNN is translation-equivariant, and geometric deep learning generalizes this to other symmetry groups. Finite fields matter for coding theory and cryptography.
Check yourself
- Can a group of order 15 have a subgroup of order 4?
Answer
No; 4 does not divide 15 (Lagrange).
- Is \(\mathbb Z/6\mathbb Z\) an integral domain?
Answer
No: \(2\cdot3=0\).
- Level
- Advanced undergraduate
- Prerequisites
- One prior exposure to groups (Landin is ideal)
- Style
- Short numbered paragraphs, exercises carry much of the theory
A concise path from set theory through Sylow theory and fields to the fundamental theorem of Galois theory, then ring theory. The cover shows a regular polygon construction, which is the classical question Galois theory answers.
Section notes
Set theory
Brisk foundations: maps, equivalence relations, cardinality, Zorn's lemma. Skim if you have read Landin.
Group theory
Group actions, orbits and stabilizers (orbit-stabilizer \(|Gx|=[G:G_x]\)), class equation, \(p\)-groups have nontrivial center. Sylow theorems: for \(|G|=p^am\) with \(p\nmid m\), Sylow \(p\)-subgroups exist, are conjugate, and their number satisfies
\[n_p\equiv1\pmod p,\qquad n_p\mid m\]
Composition series, Jordan–Hölder, solvable groups, and the structure theorem for finitely generated abelian groups.
Field theory
Algebraic extensions, splitting fields, algebraic closure, separability, normality. Finite fields: for each prime power \(p^n\) there is exactly one field \(\mathbb F_{p^n}\) up to isomorphism, the splitting field of \(x^{p^n}-x\).
Galois theory
For a finite Galois extension \(L/K\), there is an inclusion-reversing bijection between subgroups \(H\le\operatorname{Gal}(L/K)\) and intermediate fields \(L^H\), with \([L:L^H]=|H|\). Consequences: a polynomial is solvable by radicals iff its Galois group is solvable; the general quintic is not, since \(S_5\) is not solvable. Gauss's theorem: a regular \(n\)-gon is constructible iff \(n=2^kp_1\cdots p_r\) with distinct Fermat primes \(p_i\).
Ring theory
Euclidean domains ⊂ PIDs ⊂ UFDs, Gauss's lemma, \(R[x]\) is a UFD when \(R\) is. Noetherian rings and the Hilbert basis theorem.
Key results to memorize
- Orbit-stabilizer, class equation, three Sylow theorems.
- Fundamental theorem of Galois theory.
- Insolvability of the quintic and Gauss's constructibility theorem.
ML connection. Mostly indirect. Group actions and orbits are the core of equivariance and data augmentation theory; finite fields underpin error-correcting codes and hashing.
Check yourself
- Show every group of order 15 is cyclic.
Answer
\(n_3\mid5, n_3\equiv1\ (3)\Rightarrow n_3=1\); \(n_5\mid3, n_5\equiv1\ (5)\Rightarrow n_5=1\). Both Sylow subgroups are normal, so \(G\cong C_3\times C_5\cong C_{15}\).
- Is the regular 17-gon constructible?
Answer
Yes; 17 is a Fermat prime \(2^{2^2}+1\).
- Level
- Accessible survey, about 80 pages
- Prerequisites
- None
- Style
- Based on a lecture series; ideas over formalism
A tour of 20th-century logic: why foundations went into crisis, and the four landmark answers (completeness, incompleteness, undecidability, independence). Read it in a weekend, then return to it as a map when you meet these results formally.
Section notes
History and the foundational crisis
Frege's logicism, Russell's paradox \(R=\{x: x\notin x\}\), and Hilbert's program to prove mathematics consistent by finitary means.
Propositional and predicate logic
Syntax vs semantics: formulas, proofs (\(\vdash\)) and truth in a structure (\(\models\)). Quantifiers, models, and why first-order logic is the default framework.
Completeness and model theory
Gödel's completeness theorem (1930): \(\Gamma\models\varphi\iff\Gamma\vdash\varphi\). Compactness: a set of sentences has a model iff every finite subset does. Löwenheim–Skolem: a countable theory with an infinite model has a countable model, and models of every larger size. Non-standard models of arithmetic exist.
Computability
Turing machines, recursive functions, Church's thesis, and the undecidability of the halting problem by diagonalization.
Incompleteness
Gödel's first theorem: any consistent, effectively axiomatized theory containing basic arithmetic has a true-but-unprovable sentence. Second theorem: such a theory cannot prove its own consistency, which ends Hilbert's program in its original form.
Set theory and independence
ZF axioms, the axiom of choice, and the continuum hypothesis. Gödel showed CH is consistent with ZFC (constructible universe); Cohen showed its negation is too (forcing). So CH is independent.
Key results to memorize
- Completeness is about first-order logic; incompleteness is about theories of arithmetic. They do not conflict.
- Halting problem is undecidable.
- CH is independent of ZFC.
ML connection. Your knowledge-graph and ontology work sits on description logics, which are decidable fragments of first-order logic chosen precisely because full first-order reasoning is undecidable. This book explains why that trade-off exists.
Check yourself
- Why doesn't completeness contradict incompleteness?
Answer
Completeness says every sentence true in all models is provable. The Gödel sentence is true in the standard model of arithmetic but false in some non-standard model, so it is not a consequence of the axioms.
- Level
- Undergraduate, first course
- Prerequisites
- Real analysis basics (ε-δ continuity)
- Style
- Clear, motivates each abstraction from metric spaces
Six chapters from sets to the fundamental group. Mendelson's strategy is to prove things in metric spaces first, notice which proofs only used open sets, and then promote those to topological spaces.
Section notes
Theory of sets
Functions, images and preimages (note \(f^{-1}\) preserves unions, intersections and complements; \(f\) does not), countability, Cartesian products.
Metric spaces
Metrics, open balls, open and closed sets, neighborhoods. Key equivalence: \(f\) is continuous iff the preimage of every open set is open. That sentence is what makes topology possible.
Topological spaces
Topologies, bases, subspace, product and quotient topologies, closure, interior, boundary, Hausdorff spaces, homeomorphisms. A homeomorphism is a continuous bijection with continuous inverse.
Connectedness
A space is connected if it is not a union of two disjoint nonempty open sets. Continuous images of connected sets are connected, which gives the intermediate value theorem. Path-connected implies connected; the topologist's sine curve shows the converse fails.
Compactness
Every open cover has a finite subcover. Heine–Borel in \(\mathbb R^n\): compact iff closed and bounded. Continuous image of compact is compact, so a continuous real function on a compact set attains its max. In metric spaces, compact iff sequentially compact. Compact subsets of Hausdorff spaces are closed.
Homotopy and the fundamental group
Homotopic paths, \(\pi_1(X,x_0)\) as loops modulo homotopy under concatenation. \(\pi_1(\mathbb R^n)=0\), and
\[\pi_1(S^1)\cong\mathbb Z\]
via winding number. This gives the Brouwer fixed point theorem in dimension 2.
Key results to memorize
- Continuity as "preimages of open sets are open".
- Connected and compact are preserved by continuous maps.
- Heine–Borel; extreme value theorem as a corollary.
- \(\pi_1(S^1)\cong\mathbb Z\).
ML connection. Topological data analysis (persistent homology) extends the fundamental-group idea. Compactness is why minimizers exist in many optimization proofs. The "manifold hypothesis" for data is stated in this language.
Check yourself
- Is \((0,1)\) compact?
Answer
No: the cover \(\{(1/n,1)\}\) has no finite subcover.
- Why are \(\mathbb R\) and \(\mathbb R^2\) not homeomorphic?
Answer
Removing a point disconnects \(\mathbb R\) but not \(\mathbb R^2\).
- Level
- General reader to early undergraduate
- Prerequisites
- School mathematics
- Style
- Conversational, ideas first, light on proofs
A guided tour of what modern mathematics is about and why it became abstract. Use it as the shelf's orientation book: read it first, and return to the relevant chapter before starting each specialist text.
Section notes
Sets, functions, and the move to abstraction
Why mathematicians started talking about sets and maps, and how abstraction lets one proof cover many cases.
Algebra: groups and symmetry
Arithmetic mod \(n\), groups as the mathematics of symmetry, subgroups, isomorphism. Preview for Landin and Clark.
Axiomatics and counting
Axiomatic method, consistency, and infinite cardinals: \(\mathbb Q\) is countable, \(\mathbb R\) is not (Cantor's diagonal argument). Preview for Crossley.
Topology
Rubber-sheet geometry, the Möbius band, knots, topological invariants like the Euler characteristic \(V-E+F=2\) for polyhedra, and an intuitive tour of algebraic topology. Preview for Mendelson.
Linear algebra and hyperspace
Vector spaces, linear maps, and thinking in \(n\) dimensions. Preview for Shilov.
Analysis and probability
Limits, continuity and the real numbers made rigorous; probability as measure. Preview for Wilcox and Myers.
Key ideas to take away
- Structure is defined by what maps preserve: groups by homomorphisms, spaces by continuous maps.
- Different sizes of infinity exist.
- Invariants (like Euler characteristic) distinguish objects without constructing everything.
How to use it. Read one chapter as a warm-up before each specialist book. It costs an evening and tells you what the rigorous version is trying to achieve.
- Level
- Advanced undergraduate
- Prerequisites
- Real analysis (Riemann integral, uniform convergence)
- Style
- Short, self-contained, builds directly toward \(L^2\) Fourier theory
The most efficient route on the shelf to measure theory. It starts from the failures of the Riemann integral, builds Lebesgue measure on \(\mathbb R\), proves the convergence theorems, and cashes them in on Fourier series in \(L^2\).
Section notes
Limits of the Riemann integral
The Dirichlet function \(\mathbf 1_{\mathbb Q}\) on \([0,1]\) is not Riemann integrable, yet it is a pointwise limit of integrable functions. Riemann theory does not commute well with limits.
Lebesgue measure
Outer measure \(m^*(E)=\inf\sum\ell(I_k)\) over interval covers, Carathéodory measurability, the σ-algebra of measurable sets, Borel sets, sets of measure zero (every countable set), and the existence of non-measurable sets.
Measurable functions and the integral
Measurable functions, simple functions, the integral defined as a supremum over simple functions below \(f\). Riemann integrable implies Lebesgue integrable with the same value; a bounded function is Riemann integrable iff it is continuous almost everywhere.
Convergence theorems
Monotone convergence, Fatou's lemma, and the dominated convergence theorem:
\[f_n\to f \text{ a.e.},\ |f_n|\le g\in L^1\ \Rightarrow\ \int f_n\to\int f\]
DCT is the workhorse; know its hypotheses exactly.
The space \(L^2\)
Square-integrable functions modulo a.e. equality form a Hilbert space with \(\langle f,g\rangle=\int f\bar g\). Riesz–Fischer: \(L^2\) is complete.
Fourier series
Orthonormal system \(\{e^{inx}/\sqrt{2\pi}\}\), Bessel's inequality, completeness of the trigonometric system, and Parseval:
\[\frac1{2\pi}\int_{-\pi}^{\pi}|f|^2\,dx=\sum_{n\in\mathbb Z}|c_n|^2\]
Riemann–Lebesgue lemma (\(c_n\to0\)), the Dirichlet kernel and pointwise convergence for piecewise smooth \(f\), and Fejér's theorem: Cesàro means of the series of a continuous periodic function converge uniformly.
Key results to memorize
- MCT, Fatou, DCT with hypotheses.
- \(L^2\) is complete; Fourier series converge in \(L^2\) norm for every \(f\in L^2\).
- Parseval's identity; Riemann–Lebesgue lemma.
ML connection. Expectations are Lebesgue integrals, and DCT is the justification for swapping \(\nabla\) and \(\mathbb E\) in the reparameterization trick and policy gradients. Random Fourier features and spectral analysis of networks rely on the \(L^2\) theory here.
Check yourself
- What is \(\int_{[0,1]}\mathbf 1_{\mathbb Q}\,dm\)?
Answer
0, since \(\mathbb Q\cap[0,1]\) is countable and has measure zero.
- Give \(f_n\to0\) pointwise with \(\int f_n=1\).
Answer
\(f_n=n\mathbf 1_{(0,1/n)}\). No integrable dominating function exists, so DCT does not apply.
- Level
- Applied undergraduate
- Prerequisites
- Calculus, basic ODEs
- Style
- Geometric: functions as vectors; many physical problems
Davis treats Fourier series as linear algebra in function space and then uses that view to solve the heat, wave and Laplace equations, and to introduce Bessel functions and Legendre polynomials. The applied counterpart to Wilcox and Myers.
Section notes
Functions as vectors
Inner product \(\langle f,g\rangle=\int_a^b f g\,w\,dx\), orthogonality, projection onto an orthonormal set as best least-squares approximation. Fourier coefficients are just coordinates.
Fourier series
On \([-L,L]\):
\[f(x)\sim\frac{a_0}{2}+\sum_{n\ge1}\Big(a_n\cos\frac{n\pi x}{L}+b_n\sin\frac{n\pi x}{L}\Big),\quad a_n=\frac1L\int_{-L}^{L}f\cos\frac{n\pi x}{L}dx\]
Even and odd extensions, half-range sine and cosine series, complex form.
Convergence
For piecewise smooth \(f\), the series converges to the average of one-sided limits at jumps. Gibbs phenomenon: about 9% overshoot of the jump near a discontinuity, which does not shrink. Coefficient decay reflects smoothness: jumps give \(O(1/n)\), a continuous kink gives \(O(1/n^2)\).
Boundary value problems
Separation of variables for the heat equation \(u_t=ku_{xx}\), the vibrating string \(u_{tt}=c^2u_{xx}\), and Laplace's equation in rectangles and disks. Each mode evolves independently; the initial data's Fourier coefficients set the amplitudes.
Sturm–Liouville theory
Problems \((p y')'+(q+\lambda w)y=0\) with suitable boundary conditions have real eigenvalues \(\lambda_1<\lambda_2<\dots\to\infty\) and eigenfunctions orthogonal with weight \(w\). This is the spectral theorem for a differential operator.
Bessel functions and Legendre polynomials
Bessel's equation from separation in polar coordinates (vibrating drum), with modes \(J_n(\alpha_{nk}r/a)\). Legendre's equation from spherical coordinates, with orthogonal polynomials \(P_n\) on \([-1,1]\), Rodrigues' formula \(P_n=\frac1{2^nn!}\frac{d^n}{dx^n}(x^2-1)^n\).
Fourier integrals
Letting \(L\to\infty\) turns the series into the Fourier transform, for problems on unbounded domains.
Key results to memorize
- Fourier coefficients = orthogonal projections; partial sums are best \(L^2\) approximations.
- Smoothness ↔ decay rate of coefficients.
- Sturm–Liouville: real eigenvalues, orthogonal eigenfunctions.
ML connection. Sinusoidal positional encodings and RoPE are Fourier features. The "spectral bias" of neural nets (they fit low frequencies first) is described with exactly these coefficients. Legendre polynomials appear in state-space models such as HiPPO and S4.
Check yourself
- The square wave's coefficients decay like \(1/n\). What does the triangle wave's decay like, and why?
Answer
\(1/n^2\); the triangle wave is continuous with a jump only in its derivative.
- Level
- Graduate, engineering-friendly
- Prerequisites
- Advanced calculus, Fourier and Laplace transforms
- Style
- Rigorous but aimed at applications in circuits and systems
Makes the Dirac delta respectable. Distributions are continuous linear functionals on smooth test functions; with that definition you can differentiate anything, convolve, and take Fourier and Laplace transforms of objects that are not functions. The cover's spike is \(\delta\).
Section notes
Test functions and distributions
\(\mathcal D\): smooth functions with compact support, with a strong notion of convergence. A distribution \(T\in\mathcal D'\) is a continuous linear functional \(\phi\mapsto\langle T,\phi\rangle\). Every locally integrable \(f\) defines one by \(\langle f,\phi\rangle=\int f\phi\), and
\[\langle\delta,\phi\rangle=\phi(0)\]
Derivatives
Defined by moving the derivative onto the test function: \(\langle T',\phi\rangle=-\langle T,\phi'\rangle\). So every distribution is infinitely differentiable, the Heaviside step satisfies \(H'=\delta\), and differentiation commutes with limits of distributions.
Convolution
\(T*S\) under support conditions; \(\delta\) is the identity: \(\delta*T=T\), and \(\delta'*T=T'\). Differential operators become convolutions.
Fourier transform and tempered distributions
Schwartz space \(\mathcal S\) of rapidly decreasing functions is invariant under the Fourier transform, so its dual \(\mathcal S'\) is too. Results: \(\hat\delta=1\), \(\hat1=2\pi\delta\) (with the convention \(\hat f(\omega)=\int f e^{-i\omega t}dt\)), and \(\widehat{T'}=i\omega\hat T\).
Laplace transform and periodic distributions
Laplace transforms of distributions supported on \(t\ge0\), used for initial value problems. Periodic distributions have Fourier series; the Dirac comb \(\sum\delta(t-nT)\) is the basic example and underlies sampling theory.
Applications to systems
Linear time-invariant systems as convolution operators: output \(y=h*x\) with impulse response \(h\). Fundamental solutions of differential equations (\(LE=\delta\)), and passivity conditions on system transfer functions.
Key results to memorize
- \(\langle T',\phi\rangle=-\langle T,\phi'\rangle\); \(H'=\delta\).
- \(\hat\delta=1\); differentiation ↔ multiplication by \(i\omega\).
- LTI system = convolution with impulse response; solution of \(Lu=f\) is \(u=E*f\).
ML connection. Weak derivatives explain why ReLU networks train: \(\text{ReLU}'=H\), \(\text{ReLU}''=\delta\). Convolutional layers and state-space models (S4, Mamba) are LTI systems in the sense of the final chapter, with impulse response as the kernel.
Check yourself
- Compute the distributional derivative of \(|x|\) twice.
Answer
\(|x|'=\operatorname{sgn}x\), \(|x|''=2\delta\).
- Level
- Undergraduate, rigorous
- Prerequisites
- Calculus, some linear algebra
- Style
- Proves everything; linear theory first, existence theory last
The standard rigorous first course in ODEs. Coddington solves linear equations completely, handles singular points with power series, and only then proves existence and uniqueness. Start the differential-equations shelf here.
Section notes
Preliminaries and first-order linear equations
Complex-valued functions, \(e^{(a+ib)x}\). The equation \(y'+a(x)y=b(x)\) solved with an integrating factor \(e^{\int a}\).
Linear equations with constant coefficients
\(L(y)=y^{(n)}+a_1y^{(n-1)}+\dots+a_ny\). Characteristic polynomial \(p(r)\); root \(r\) of multiplicity \(m\) gives \(e^{rx},xe^{rx},\dots,x^{m-1}e^{rx}\). The Wronskian tests independence. Nonhomogeneous equations by annihilators or variation of parameters.
Linear equations with variable coefficients
Existence of a basis of \(n\) solutions, Abel's formula \(W(x)=W(x_0)\exp\!\big(-\!\int_{x_0}^x a_1\big)\), reduction of order, variation of parameters. Legendre's equation solved by power series at an ordinary point.
Regular singular points
Euler equations \(x^2y''+axy'+by=0\), then the Frobenius method: try \(y=x^r\sum c_kx^k\), solve the indicial equation; roots differing by an integer can force a logarithm. Bessel's equation as the main example.
Existence and uniqueness: first order
For \(y'=f(x,y)\) with \(f\) continuous and Lipschitz in \(y\), Picard iteration
\[\phi_{k+1}(x)=y_0+\int_{x_0}^x f(t,\phi_k(t))\,dt\]
converges uniformly to the unique solution on a small interval. Without Lipschitz, uniqueness can fail: \(y'=3y^{2/3}\), \(y(0)=0\).
Existence and uniqueness: systems
The same proof in vector form for \(\mathbf y'=\mathbf f(x,\mathbf y)\), which covers \(n\)th-order equations. Linear systems have solutions on the whole interval where coefficients are continuous.
Key results to memorize
- Solution space of an \(n\)th-order linear homogeneous ODE is \(n\)-dimensional.
- Abel's formula; Frobenius indicial equation.
- Picard–Lindelöf: Lipschitz gives local existence and uniqueness.
ML connection. Neural ODEs and diffusion-model samplers are ODE solvers; their well-posedness is Picard–Lindelöf with a Lipschitz network. Residual blocks \(x_{k+1}=x_k+f(x_k)\) are Euler steps.
Check yourself
- Solve \(y''-2y'+y=0\).
Answer
Double root \(r=1\): \(y=(c_1+c_2x)e^x\).
- Why does \(y'=3y^{2/3}, y(0)=0\) have two solutions?
Answer
\(y=0\) and \(y=x^3\); \(f\) is not Lipschitz near \(y=0\).
- Level
- Early undergraduate
- Prerequisites
- Algebra; calculus helpful
- Style
- Applications from economics, psychology and sociology
The discrete twin of ODEs. Goldberg builds the calculus of finite differences, solves linear difference equations, and uses them to model markets, learning and populations. Probably the most directly useful book on the shelf for an ML engineer.
Section notes
Calculus of finite differences
Difference operator \(\Delta y_k=y_{k+1}-y_k\), shift \(Ey_k=y_{k+1}\), so \(\Delta=E-1\). Factorial powers \(k^{(n)}=k(k-1)\cdots(k-n+1)\) satisfy \(\Delta k^{(n)}=nk^{(n-1)}\), the discrete power rule. Summation is anti-differencing.
First-order linear equations
\(y_{k+1}=ay_k+b\) has solution
\[y_k=a^k\Big(y_0-\frac{b}{1-a}\Big)+\frac{b}{1-a}\quad(a\ne1)\]
Converges to the equilibrium iff \(|a|<1\); oscillates if \(a<0\). Application: the cobweb model of supply and demand.
Higher-order linear equations
Constant coefficients: try \(y_k=\lambda^k\), solve the characteristic equation. Repeated root gives \(k\lambda^k\); complex roots \(re^{\pm i\theta}\) give \(r^k\cos k\theta\), \(r^k\sin k\theta\) (damped oscillation if \(r<1\)). Particular solutions by undetermined coefficients.
Stability
Equilibrium is stable iff every characteristic root has \(|\lambda|<1\). For \(\lambda^2+a_1\lambda+a_2=0\) this is
\[1+a_1+a_2>0,\quad 1-a_1+a_2>0,\quad 1-a_2>0\]
Applications
Samuelson's multiplier-accelerator: \(Y_t=c(1+v)Y_{t-1}-cvY_{t-2}+G\), producing growth, decay or business cycles depending on \(c,v\). Harrod–Domar growth, stochastic learning models in psychology, and population and diffusion models in sociology.
Key results to memorize
- \(\Delta=E-1\); factorial powers behave like ordinary powers under \(\Delta\).
- Stability ⇔ all characteristic roots inside the unit circle.
- Jury conditions for second order.
ML connection. SGD with momentum on a quadratic is a second-order linear difference equation: \(x_{k+1}=x_k-\eta\lambda x_k+\beta(x_k-x_{k-1})\). Apply the stability conditions above to get the exact range of stable \((\eta,\beta)\). RNN state updates and exponential moving averages (Adam's moments) are first-order difference equations; vanishing and exploding gradients are \(|\lambda|<1\) vs \(|\lambda|>1\).
Check yourself
- Solve \(y_{k+2}-y_{k+1}-y_k=0\).
Answer
\(y_k=c_1\varphi^k+c_2\psi^k\), \(\varphi,\psi=(1\pm\sqrt5)/2\). Fibonacci.
- For gradient descent \(x_{k+1}=(1-\eta\lambda)x_k\), when does it converge?
Answer
\(|1-\eta\lambda|<1\iff0<\eta<2/\lambda\).
- Level
- Undergraduate, the best first PDE book here
- Prerequisites
- ODEs (Coddington), multivariable calculus
- Style
- Careful, honest about convergence; strong complex-variables part
Weinberger covers the three classical equations, eigenfunction expansions, and a full treatment of complex variables and transform methods. Read it as your first PDE book.
Section notes
The one-dimensional wave equation
d'Alembert's solution of \(u_{tt}=c^2u_{xx}\):
\[u(x,t)=\tfrac12\big[f(x-ct)+f(x+ct)\big]+\frac1{2c}\int_{x-ct}^{x+ct}g(s)\,ds\]
Characteristics \(x\pm ct=\)const, domain of dependence, finite speed of propagation, reflections at boundaries.
Classification and characteristics
Second-order equations \(Au_{xx}+2Bu_{xy}+Cu_{yy}+\dots\) are hyperbolic, parabolic or elliptic as \(B^2-AC\) is \(>0\), \(=0\), \(<0\). Which data give a well-posed problem depends on type.
Separation of variables and Fourier series
Product solutions, eigenvalue problems, convergence of the resulting series, and verifying that the series actually solves the PDE.
Laplace's equation and the maximum principle
Harmonic functions attain their max and min on the boundary, which gives uniqueness for the Dirichlet problem. Poisson's integral formula for the disk.
Eigenfunction expansions
Sturm–Liouville problems and their multidimensional analogs, completeness, and expansions in Bessel and Legendre functions.
Green's functions
Solve \(Lu=f\) as \(u(x)=\int G(x,\xi)f(\xi)\,d\xi\), where \(G\) solves the equation with a point source and the homogeneous boundary conditions.
Complex variables and conformal mapping
Analytic functions, Cauchy's theorem and integral formula, residues. Real and imaginary parts of analytic functions are harmonic, so conformal maps carry solutions of Laplace's equation between domains.
Fourier and Laplace transforms
Transforms turn derivatives into multiplication; inversion by contour integration. Heat equation on the line gives the Gaussian kernel \(u=\frac1{\sqrt{4\pi kt}}e^{-x^2/4kt}*f\).
Key results to memorize
- d'Alembert's formula; classification by \(B^2-AC\).
- Maximum principle ⇒ uniqueness.
- Heat kernel is a Gaussian with variance \(2kt\).
ML connection. The heat kernel is the forward process of a diffusion model: adding Gaussian noise solves the heat equation. Graph Laplacians in GNNs are discrete versions of \(\Delta\), and physics-informed neural networks train on exactly these equations.
Check yourself
- Classify \(u_{xx}+xu_{yy}=0\).
Answer
\(B^2-AC=-x\): elliptic for \(x>0\), hyperbolic for \(x<0\), parabolic on \(x=0\) (Tricomi equation).
- Level
- Undergraduate to early graduate
- Prerequisites
- Advanced calculus, ODEs
- Style
- Geometric, strongest on first-order equations and characteristics
Distinctive for its thorough geometric treatment of first-order equations and characteristic surfaces before reaching the classical second-order theory. Read it after Weinberger to deepen the characteristics viewpoint.
Section notes
Curves, surfaces and vector fields
Integral curves of vector fields and first integrals: the geometry that first-order PDEs are really about.
First-order linear and quasilinear equations
For \(a(x,y,u)u_x+b(x,y,u)u_y=c(x,y,u)\), solutions are surfaces made of characteristic curves
\[\frac{dx}{a}=\frac{dy}{b}=\frac{du}{c}\]
The Cauchy problem is solvable near an initial curve that is not characteristic. Shock formation in nonlinear equations like \(u_t+uu_x=0\).
Fully nonlinear first-order equations
Monge cones, characteristic strips, and complete integrals for \(F(x,y,u,p,q)=0\). Connection to Hamilton–Jacobi equations.
Cauchy–Kovalevskaya and classification
Analytic Cauchy problems have unique local analytic solutions. Second-order classification, canonical forms, and characteristic surfaces in higher dimensions.
Laplace, wave and heat equations
Harmonic functions and the mean value property, the wave equation in 1, 2 and 3 dimensions (Huygens' principle holds in odd dimensions ≥ 3), and the heat equation with its maximum principle and smoothing.
Key results to memorize
- Method of characteristics for quasilinear first-order equations.
- Non-characteristic initial data is required for the Cauchy problem.
- Huygens' principle holds in 3D, fails in 2D.
ML connection. Hamilton–Jacobi–Bellman equations in optimal control and RL are first-order nonlinear PDEs; characteristics correspond to optimal trajectories. Score-based diffusion uses the continuity (transport) equation, another first-order PDE.
Check yourself
- Solve \(u_t+cu_x=0\), \(u(x,0)=f(x)\).
Answer
Characteristics \(x-ct=\)const, so \(u=f(x-ct)\).
- Level
- Advanced undergraduate to graduate
- Prerequisites
- Advanced calculus, a first PDE course
- Style
- Classic Moscow lectures; organized by equation type
A classic organized around the three types of second-order equation. Its elliptic chapter includes the Fredholm theory of integral equations, which is the bridge from classical PDE to functional analysis.
Section notes
Introduction and classification
Classification, canonical forms, the Cauchy–Kovalevskaya theorem, and Hadamard's example showing analytic solvability is not enough: for Laplace's equation, small Cauchy data can produce huge solutions, so the problem is ill-posed.
Hyperbolic equations
The Cauchy problem for the wave equation, energy integrals and uniqueness, domains of dependence. Kirchhoff's formula in 3D and Poisson's in 2D (method of descent).
Elliptic equations
Harmonic functions, Dirichlet and Neumann problems, Green's function, single- and double-layer potentials. Reducing boundary problems to integral equations, then the Fredholm alternative:
\[\phi-\lambda K\phi=f\ \text{ solvable for all } f\iff \phi-\lambda K\phi=0 \text{ has only } \phi=0\]
Parabolic equations
The heat equation, maximum principle, uniqueness, and the Cauchy problem solved by the heat kernel.
Key results to memorize
- Hadamard well-posedness: existence, uniqueness, continuous dependence.
- Energy method for hyperbolic uniqueness.
- Fredholm alternative.
ML connection. Ill-posedness is the core issue in inverse problems and in learning operators from data; regularization exists to restore continuous dependence.
- Level
- Advanced undergraduate to graduate
- Prerequisites
- Advanced calculus; a first PDE course helps
- Style
- Thorough and physically motivated; integral equations given full weight
A comprehensive treatment of the equations of physics with a long integral-equations component and an introduction to variational methods and distributions. Use it as the reference volume of the PDE shelf.
Section notes
Derivations and the wave equation
Physical derivations of the classical equations, then the wave equation in one and several dimensions, energy, and Huygens' principle.
Fourier series and integrals
Convergence theory and transform methods applied to boundary and initial value problems.
The heat equation
Maximum principle, the heat kernel on \(\mathbb R^n\), and problems on bounded domains.
Laplace's equation and potential theory
Mean value property, Harnack's inequality, Green's functions, Newtonian potentials, and \(\Delta\big(\tfrac{-1}{4\pi|x|}\big)=\delta\) in \(\mathbb R^3\).
Integral equations
Fredholm equations of the second kind, the Fredholm alternative, symmetric kernels and the Hilbert–Schmidt theorem: a symmetric compact kernel has an eigenfunction expansion \(K(x,y)=\sum\lambda_n\phi_n(x)\phi_n(y)\) in the \(L^2\) sense. Volterra equations always have unique solutions.
Variational methods and eigenvalues
Dirichlet's principle, weak solutions, and Rayleigh–Ritz approximation of eigenvalues by minimizing the Rayleigh quotient.
Key results to memorize
- Fundamental solution of the Laplacian.
- Hilbert–Schmidt expansion for symmetric kernels.
- Eigenvalues as minima of Rayleigh quotients.
ML connection. The Hilbert–Schmidt expansion is the infinite-dimensional version of Mercer's theorem, the foundation of kernel methods and Gaussian processes. Kernel PCA is literally solving an integral eigenproblem.
- Level
- Advanced undergraduate to graduate
- Prerequisites
- A first PDE course, linear algebra, some real analysis
- Style
- Three parts: classical PDE, Hilbert space methods, computation
The modern view of PDEs: solutions live in function spaces, operators have spectra, and well-posedness follows from abstract theorems. This is where the analysis books and the PDE books meet.
Section notes
Classical PDE review
Laplace, heat and wave equations, separation of variables, Green's functions, characteristics. Moves quickly if you have read Weinberger.
Hilbert spaces and distributions
\(L^2\), orthogonal projection, Riesz representation, generalized functions and weak derivatives.
Sobolev spaces and weak solutions
\(H^1_0(\Omega)\): functions vanishing on the boundary with square-integrable weak gradient. The weak form of \(-\Delta u=f\) is
\[\int_\Omega\nabla u\cdot\nabla v\,dx=\int_\Omega fv\,dx\quad\forall v\in H^1_0\]
Lax–Milgram: a bounded coercive bilinear form gives a unique weak solution.
Operator and spectral theory
Unbounded operators, self-adjointness, compact operators, and the spectral theorem in infinite dimensions. Eigenfunction expansions become theorems about self-adjoint operators with compact resolvent.
Numerical methods
Finite differences and finite elements. FEM is the weak form restricted to a finite-dimensional subspace (Galerkin), so error analysis comes straight from the Hilbert space theory.
Key results to memorize
- Weak formulation and Lax–Milgram.
- Galerkin approximation is orthogonal projection in the energy norm.
- Compact self-adjoint operators have orthonormal eigenbases.
ML connection. Reproducing kernel Hilbert spaces, neural operators (FNO, DeepONet) and the neural tangent kernel are all phrased in this Hilbert-space language. Sobolev norms show up as smoothness regularizers.
- Level
- Advanced undergraduate to graduate
- Prerequisites
- Advanced calculus, complex variables
- Style
- Classical analysis with integral equations and analytic methods
A compact, analytic treatment that leans on integral equations, complex analysis and analytic function theory, closing with topics from the author's research area of scattering theory. Read late on the ladder.
Section notes
Preliminaries
Fourier series, integral transforms, and an introduction to integral equations as tools used throughout.
First-order equations and classification
Characteristics, the Cauchy problem, the Cauchy–Kovalevskaya theorem, and reduction of second-order equations to canonical form.
The heat equation
Initial and boundary value problems, the heat kernel, maximum principle, and integral-equation methods for boundary problems.
Laplace's equation and potential theory
Green's functions, layer potentials, and Fredholm theory applied to the Dirichlet and Neumann problems.
The wave and Helmholtz equations
Time-harmonic waves \(u(x,t)=\mathrm{Re}\{v(x)e^{-i\omega t}\}\) reduce the wave equation to \(\Delta v+k^2v=0\). Radiation conditions, scattering by obstacles, and the setup of inverse scattering problems.
Key results to memorize
- Helmholtz equation from separating time.
- Layer potentials turn boundary problems into integral equations on the boundary.
ML connection. Inverse scattering is an inverse problem; learned regularizers and physics-informed networks are current approaches to it.
- Level
- Advanced undergraduate to graduate survey
- Prerequisites
- ODEs, linear algebra, some analysis
- Style
- Broad catalogue of nonlinear problems and solution methods
A wide survey of nonlinear equations of every kind (algebraic, difference, differential, integral, functional) organized around a few methods: iteration and fixed points, linearization, perturbation, and qualitative analysis. Read for breadth after Goldberg and Coddington.
Section notes
Nonlinear algebraic and operator equations
Newton's method \(x_{k+1}=x_k-J(x_k)^{-1}F(x_k)\) with quadratic convergence near a simple root. Contraction mapping theorem: if \(\|T x-Ty\|\le q\|x-y\|\), \(q<1\), on a complete space, then \(T\) has a unique fixed point and iteration converges geometrically. Brouwer and Schauder fixed point theorems for existence without uniqueness.
Nonlinear difference equations
Fixed points and their stability via \(|f'(x^*)|<1\); period doubling in maps like the logistic map \(x_{k+1}=rx_k(1-x_k)\).
Nonlinear ordinary differential equations
Phase plane, linearization at equilibria (node, saddle, focus, center), Lyapunov functions for stability, limit cycles and the Poincaré–Bendixson theorem, perturbation and averaging methods.
Nonlinear integral and functional equations
Hammerstein and Urysohn integral equations solved by iteration or fixed-point arguments; equations where the unknown is a function of transformed arguments.
Nonlinear partial differential equations
Examples such as Burgers' equation (solvable by the Cole–Hopf transform into the heat equation) and the role of similarity solutions.
Key results to memorize
- Banach contraction mapping theorem.
- Stability of equilibria from eigenvalues of the Jacobian (Hartman–Grobman when hyperbolic).
- Lyapunov's direct method.
ML connection. Training dynamics are a nonlinear dynamical system; stability at a minimum is the eigenvalue analysis here. Deep equilibrium models and fixed-point iteration in transformers use the contraction mapping theorem directly.
- Level
- Advanced undergraduate
- Prerequisites
- ODEs, complex variables helpful
- Style
- Classical and explicit; special functions and worked models
A hands-on catalogue of nonlinear equations that can be solved or analyzed explicitly: Riccati equations, elliptic functions, Painlevé transcendents, nonlinear oscillators and population models. The spiral on the cover is a trajectory approaching a focus in the phase plane.
Section notes
Riccati equation
\(y'=P(x)+Q(x)y+R(x)y^2\). The substitution \(y=-u'/(Ru)\) turns it into a linear second-order equation. Knowing one solution reduces it to a linear first-order one.
Elliptic functions and the pendulum
\(\theta''+\frac gL\sin\theta=0\) is solved by Jacobi elliptic functions. The period with amplitude \(\theta_0\) is
\[T=4\sqrt{L/g}\;K(k),\qquad k=\sin(\theta_0/2)\]
which reduces to \(2\pi\sqrt{L/g}\) for small swings.
Painlevé transcendents
Second-order equations whose only movable singularities are poles reduce to known functions or to six new ones, the Painlevé transcendents. Example: \(y''=6y^2+x\) (P-I).
Nonlinear oscillations and the phase plane
Duffing \(x''+\delta x'+\alpha x+\beta x^3=\gamma\cos\omega t\) (amplitude-dependent frequency, jump phenomena) and van der Pol \(x''-\mu(1-x^2)x'+x=0\) (a stable limit cycle). Singular points classified by linearization.
Population models
Logistic growth \(N'=rN(1-N/K)\) and Volterra's predator-prey system with closed orbits around the coexistence equilibrium.
Nonlinear integral equations and numerics
Volterra-type nonlinear integral equations, existence by successive approximation, and numerical integration of nonlinear ODEs.
Key results to memorize
- Riccati ↔ linear second-order via \(y=-u'/(Ru)\).
- Pendulum period grows with amplitude through \(K(k)\).
- Van der Pol has a unique stable limit cycle for \(\mu>0\).
ML connection. The Riccati equation is central to LQR control and Kalman filtering. Limit cycles and oscillations show up in GAN training dynamics, which is a two-player system like predator-prey.