The 30 Papers

Course notes that build convolutional networks from linear classifiers, optimization, and backpropagation through the architectures used for visual recognition.

Why it matters Start here when the notation or training loop in later papers feels compressed. The notes supply the working vocabulary that those papers assume.

A character-level recurrent network learns enough local and long-range structure to generate convincing prose, code, and mathematical text from raw sequences.

Why it matters The examples make sequence modeling tangible before the formal papers arrive, including both the surprising capability and the failure modes of recurrent generation.

A visual account of the LSTM cell, showing how gates regulate what a recurrent network keeps, writes, exposes, and forgets across a sequence.

Why it matters Read it before the sequence papers so their LSTM equations resolve into a mechanism rather than a wall of symbols.

AlexNet combines a large labeled dataset, GPU training, rectified linear units, dropout, and data augmentation to decisively improve ImageNet classification.

Why it matters The paper marks the practical break between hand-engineered vision pipelines and the modern deep-learning recipe of scale, compute, and learned representation.

Residual blocks learn changes relative to an identity path, making optimization stable enough to train image networks hundreds of layers deep.

Why it matters Skip connections became a durable architectural primitive far beyond vision. This paper shows the optimization problem they were designed to solve.

Dilated convolutions enlarge a network's receptive field without reducing spatial resolution, allowing dense predictions to use context at several scales.

Why it matters It is a compact study in changing an operator to preserve information that pooling would discard, a recurring concern in structured prediction.

Clean identity shortcuts and pre-activation blocks improve information and gradient flow through very deep residual networks.

Why it matters The follow-up explains which part of ResNet matters and why. It rewards reading the original architecture as a hypothesis that can be tested and refined.

Dropout regularizes large LSTMs when applied to their non-recurrent connections, improving generalization without disrupting the recurrent state transition.

Why it matters The paper separates a useful training intervention from the places where that intervention breaks temporal memory, a practical lesson in respecting architecture.

A single end-to-end system combines recurrent networks, convolution, CTC training, and large-scale compute for speech recognition in two languages.

Why it matters It shows what changes when a model leaves a benchmark and becomes a production-scale sequence system: data, optimization, latency, and hardware all enter the argument.

Sequence models impose an order even when their inputs or outputs are sets, and the chosen order can materially change what the model learns.

Why it matters The paper exposes a representational mismatch that is easy to ignore: the machinery's sequence bias can leak into a problem whose structure is permutation-invariant.

A translation decoder learns a soft alignment over source words at each output step instead of relying on one fixed-length encoded sentence.

Why it matters This is attention before the Transformer: a direct answer to the bottleneck created when an entire source sequence must fit into one vector.

The decoder's attention distribution becomes the output, allowing a network to point to positions in an input whose length varies from example to example.

Why it matters The paper turns attention from an internal weighting mechanism into an interface for structured prediction, extending what a sequence model can name.

The Transformer replaces recurrence with multi-head self-attention, positional information, and feed-forward layers that train efficiently in parallel.

Why it matters Most modern language-model architecture starts here. Read it after the earlier attention papers to see which constraints the Transformer removes and which it keeps.

A runnable implementation walks through the Transformer line by line, connecting the paper's equations to tensors, masks, optimization, and training code.

Why it matters Use it as the laboratory companion to Attention Is All You Need. The implementation turns architecture diagrams into operations you can inspect.

A neural controller learns differentiable read and write operations over an external memory, acquiring simple algorithms from input-output examples.

Why it matters The work asks what changes when memory becomes an addressable component rather than a hidden state, a question still active in agent systems.

Relation Networks apply a shared function to pairs of object representations, giving a larger system an explicit mechanism for comparing entities.

Why it matters The module is intentionally small. Its value comes from matching the architecture to the structure of relational questions rather than adding undirected depth.

A recurrent core uses self-attention over a memory matrix so stored items can interact before the next state is produced.

Why it matters It joins two threads in the list, recurrence and relational attention, and tests whether memory works better when its contents can reason over one another.

A common message-passing framework unifies several graph neural networks and predicts molecular properties by exchanging information along chemical bonds.

Why it matters The framework makes relational computation concrete on graphs, where the data structure already declares which entities may communicate.

Language-model loss follows predictable power laws with model size, data, and compute across the measured regime, exposing tradeoffs between those inputs.

Why it matters The paper turns scaling from an intuition into an empirical planning model and helps explain why research strategy moved toward larger training runs.

Pipeline parallelism partitions a model across accelerators and splits batches into micro-batches so several partitions can work at once.

Why it matters Scaling laws only matter when systems can execute the implied training runs. GPipe shows one mechanism for turning model size into a schedulable workload.

A network can be regularized by minimizing how many bits are needed to describe its weights while still accounting for the training data.

Why it matters The paper connects generalization to compression early in the history of neural networks and sets up the description-length readings that follow.

Minimum Description Length chooses the model that gives the shortest combined encoding of the model and the data explained by it.

Why it matters The tutorial supplies the formal language behind the intuition that learning and compression are closely related, without requiring the reader to infer it from fragments.

A blog essay asks how to formalize the observation that complexity often rises and later falls as a closed system approaches equilibrium.

Why it matters The question reframes complexity as a trajectory rather than a synonym for entropy, preparing the concrete coffee-automaton model that follows.

A cellular automaton for mixing coffee and cream tests candidate measures of complexity as a system moves from order through structure to equilibrium.

Why it matters The model gives the previous essay an object you can simulate, measure, and disagree with instead of leaving the complexity claim at the level of metaphor.

Kolmogorov complexity defines the information in a string by the length of the shortest program that can generate it on a fixed universal machine.

Why it matters This is the formal backbone of algorithmic description length, including the sharp limit that the quantity itself cannot generally be computed.

An autoregressive decoder and variational latent code divide information so the model can choose what to preserve globally and what to reconstruct locally.

Why it matters The paper makes lossy representation a design variable, connecting the abstract compression block to a trainable generative model.

A doctoral thesis develops a formal measure of universal intelligence and studies the behavior and consequences of agents that score highly under it.

Why it matters It closes the path by asking what intelligence means when the definition must range across environments rather than one benchmark or one task family.