Concept notes

The determinant of a matrix is the signed volume scaling factor of the linear map. Zero determinant means the map collapses dimensions.

Eigenvectors are directions a matrix only stretches. The spectral theorem says symmetric matrices have a full orthogonal eigenbasis with real eigenvalues.

A matrix is a linear function from one vector space to another. Every operation in ML. Projection, rotation, basis change, gradient flow. Is matrix multiplication.

Gradients, Jacobians, and Hessians for vector- and matrix-valued functions. The minimum needed to derive backprop and second-order methods.

Matrices that define inner products and proper covariances. The geometry of PSD: ellipsoids, not arbitrary shapes.

The singular value decomposition factorizes any matrix into rotation × stretching × rotation. PCA is SVD applied to mean-centered data.

How to update beliefs given evidence: posterior ∝ likelihood × prior. The foundation of Bayesian inference, naive Bayes, and probabilistic graphical models.

An estimator has bias (systematic error) and variance (sample-to-sample wobble). Mean-squared error decomposes into the two.

Estimate uncertainty by resampling the observed units. Preserve pairing and dependence, report the resampling unit, and know when the bootstrap can fail.

Prediction estimates what will happen; causal inference estimates what an intervention will change. Define the estimand, assumptions, assignment process, and identification strategy.

Sums of many independent random variables become Gaussian. Why nearly every error bar in ML and statistics is computed from a normal distribution.

Entropy measures uncertainty, while mutual information measures shared dependence. Both require careful estimation and neither proves causation or calibrated confidence.

Epistemic uncertainty shrinks with more data; aleatoric uncertainty does not. Confusing them causes miscalibration and wasted data collection.

Use moments to describe location, uncertainty, and dependence. Know what covariance measures, how transformations change it, and why correlation does not establish causation.

A unified family of distributions (Gaussian, Bernoulli, Poisson, Beta, Gamma, etc.) with shared properties: sufficient statistics, conjugate priors, simple MLE.

Use uncertainty, effect size, and test assumptions to decide whether a measured model gain is likely to be real.

Asymmetric distance between probability distributions. Cross-entropy minus entropy. The mathematical glue holding most of probabilistic ML together.

Stochastic processes where the future depends only on the present, not the past. Foundation of HMMs, MCMC, and many sequence models.

The dominant statistical principle: pick parameters that make the observed data most probable. Reduces to minimizing cross-entropy for classification and MSE for Gaussian regression.

Estimate expectations by averaging over random samples. The simplest way to compute integrals you can't compute analytically.

Choose a distribution from the data type and generation process. Connect its support, parameters, mean, variance, likelihood, and common ML use.

Density-based clustering: form clusters from regions of high point density, label sparse points as noise. Handles arbitrary cluster shapes; no k to specify.

Recursively split the feature space along axis-aligned thresholds chosen to maximize a purity criterion. The base learner of GBDT and random forests.

Train trees sequentially, each one fitting the gradient of the loss with respect to the current ensemble's prediction. The dominant tabular learner in 2026.

Partition n points into k clusters by minimizing within-cluster variance. Lloyd's algorithm: alternate assigning points to nearest center and recomputing centers.

Compute inner products in a high-dimensional feature space without ever materializing the features. The mathematical move that lets a linear classifier draw nonlinear boundaries.

Predict a continuous target as a linear combination of features by minimizing squared error. Closed-form solution, MLE under Gaussian noise, and the foundation everything else builds on.

Linear regression for binary classification: pass a linear combination through a sigmoid, train by maximum likelihood. Still the strongest non-trivial baseline for tabular classification.

A trivially simple generative classifier that assumes features are conditionally independent given the class. Fast, parameter-light, surprisingly hard to beat on text.

Bag deep decision trees plus random feature subsets per split. Variance averaging beats any single tree; the dominant out-of-the-box ensemble before GBDT.

Maximum-margin classifier with a kernel that lets it operate in implicit high-dimensional feature spaces. Beautiful theory; less common in 2026 production.

Both project high-dimensional data to 2D for visualization by preserving local neighborhoods. Both are easy to misread. Know what they show and what they hide.

ReLU, GELU, swish, sigmoid, tanh. What each does, why GELU/swish replaced ReLU in transformers, and when to use which.

Two paradigms for generative modeling: predict the next element step-by-step (autoregressive) or iteratively denoise from pure noise (diffusion). Different costs, different strengths.

Reverse-mode automatic differentiation applied to a computation graph. The algorithm that computes gradients for every parameter in one backward pass.

Learn useful representations from unlabeled data by defining which views should agree and which examples should stay apart.

An encoder summarizes the input into a representation; a decoder generates the output conditioned on it. The structure behind translation, T5, summarization, and many multimodal models.

Why deep networks were untrainable before residuals, normalization, and ReLU. The math of gradient magnitudes through depth and the standard fixes.

Neighbors carry signal. A graph neural network averages each node's neighborhood and projects with a learned matrix. The same matmul as a CNN, on irregular structure.

Add the input of a block to its output. Lets gradients flow unimpeded through depth and made networks deeper than 30 layers practical for the first time.

Compute a weighted sum of values, weights derived from query-key similarity. The single operation that powers transformers, retrieval, and most of modern ML.

A neural network with one hidden layer and enough units can approximate any continuous function on a bounded domain. What it does and doesn't say about deep learning.

Learn to invert a fixed noising process. The dominant generative paradigm for images, audio, video, and molecules in 2026.

How to get gradients through a sampling step over discrete variables, where the reparameterization trick doesn't apply. Covers the score-function (REINFORCE) estimator, the straight-through estimator, and Gumbel-Softmax.

Two networks compete: a generator produces samples, a discriminator distinguishes them from real data. Sharp samples, training instability, mostly displaced by diffusion in 2026.

Generative models built from invertible transformations. Compute exact likelihoods and sample efficiently. At the cost of architectural restrictions.

Encode inputs to a latent distribution, decode samples back, optimize evidence lower bound. The cleanest gateway to deep generative models.

Belief propagation computes graphical-model marginals through local messages. Sum-product is exact on trees and approximate on graphs with cycles.

Iterate between estimating latent variables given parameters (E-step) and updating parameters given latents (M-step). The standard tool for latent-variable MLE when the latents are unobserved.

Factor analysis uses latent factors with per-feature noise. Probabilistic PCA uses isotropic noise and recovers classical PCA in its zero-noise limit.

Sum and max over exponentially many paths in linear time. Forward-backward computes posteriors over hidden states; Viterbi finds the most likely state sequence. The same idea, two semirings.

Model data as a weighted sum of K Gaussians. Soft clustering, density estimation, and the canonical EM example.

A distribution over functions defined by a mean and covariance kernel. Predicts a mean and kernel-conditional uncertainty. Beautiful theory, brutal scaling.

A latent Markov chain emits observations through a per-state distribution. Forward-backward, Viterbi, Baum-Welch. The classical sequence model toolkit.

Express joint distributions as graphs whose structure encodes conditional independence. Bayesian networks (directed) and Markov random fields (undirected).

Policy gradient with a learned value baseline. The actor picks actions; the critic estimates how good they were. The architecture under PPO, A3C, SAC, and most modern RL.

Policy gradients need a low-variance estimate of how much better an action was than average. GAE is the standard answer: an exponentially weighted blend of n-step returns.

Choose actions from context while balancing reward and uncertainty. The bridge between supervised prediction, experimentation, and reinforcement learning.

An RL or bandit agent has to keep trying new actions to learn while taking the best-known action to score. Three classical strategies, each with a different way of resolving the tension.

Define sequential decisions with states, actions, transitions, rewards, and value functions before choosing an RL algorithm.

Learning when other agents change the environment: non-stationarity, credit assignment, coordination, competition, and evaluation.

Directly optimize the policy by following the gradient of expected return. REINFORCE, actor-critic, and the foundation of modern RL.

Constrain policy updates with a clipped surrogate objective. The default actor-critic algorithm in 2026. For robotics, games, and RLHF.

Learn the action-value function Q(s, a) by Bellman backups. The foundation of value-based RL. DQN, Rainbow, and the original Atari breakthroughs.

Modify learning signals without accidentally changing the task, creating reward hacking, or hiding specification failure.

The environment defines what behavior is possible; the grader defines what optimization values. Both need versioning, adversarial tests, and evidence.

Train a model from outcomes that can be checked, using grouped samples to estimate relative advantage without a separate value model.

Learn actions from demonstrations, rewards, or world models while respecting partial observability, control frequency, safety, and sim-to-real shift.

Two paradigms in reinforcement learning. Value-based learns Q(s, a) and acts greedily; policy-based directly parametrizes the policy. When to use which.

Small worst-case perturbations, threat models, adversarial training, robust evaluation, and the difference between security and ordinary distribution shift.

Object detectors predict thousands of overlapping boxes. Anchors give each prediction a prior shape; NMS prunes near-duplicates. The pre-DETR pipeline that defined the field for a decade.

Convolutions encode translation equivariance and locality. The structural inductive bias that powered the deep learning revolution in vision.

A 2D convolution is a matmul in disguise. Unfold the input into columns, multiply by a flattened filter matrix. The reason CNNs run fast on the same hardware as transformers.

Transfer a model across related but shifted data distributions without assuming unlabeled target data makes the problem identifiable.

Align modality encoders, token budgets, fusion, objectives, and evaluation without pretending text, image, audio, and video share one natural representation.

Localize and classify objects in an image. The three main architectural families: two-stage proposal-based, one-stage grid-based, and transformer-based.

Residual connections enabled networks deeper than 30 layers to train. Still the dominant backbone for transfer learning in 2026.

Assign a class to every pixel: encoder-decoder architectures, losses, IoU, class imbalance, boundaries, and deployment constraints.

Apply a standard transformer to a sequence of image patches. Beats CNNs at scale; the dominant backbone for foundation vision models in 2026.

The end-to-end map from a waveform to text: features, the three modeling paradigms (CTC, RNN-T, attention), language-model fusion, and how the field moved from HMM-GMM pipelines to a single neural model.

Train a transformer to fill in randomly masked tokens. The result is a bidirectional encoder that broke a dozen NLP benchmarks at once and defined the pretrain-then-finetune era.

A CRF models labels for a whole sequence and scores transitions jointly. Linear-chain CRFs improve taggers when neighboring labels constrain each other.

How you train a sequence model to map audio (or pixels) to text without knowing the alignment. CTC marginalizes over every possible alignment with a blank symbol and a forward-backward sum.

Compare modular acoustic-pronunciation-language pipelines with CTC, attention, and transducer systems across data, control, latency, and operations.

Recurrent networks fail because gradients vanish through repeated matmul. Gates fix this by using elementwise multiplication to control information flow. Then transformers replaced them anyway.

The streaming-ASR workhorse. RNN-T fixes CTC's biggest weakness (its frame-independence assumption) by adding a prediction network that conditions on previously emitted tokens, while staying naturally streamable.

Speaker verification and identification using embeddings, metric learning, calibration, anti-spoofing, and operating-point evaluation.

Emit transcripts with bounded latency using chunked encoders, monotonic alignment, endpointing, and stability-aware evaluation.

Map words to dense vectors so that similar words land near each other. The breakthrough that proved meaning lives in geometry, not symbols.

Factorize the user-item matrix into two low-rank factors. Each is a linear regression given the other, so alternate. The classical recsys workhorse before deep learning.

Exact k-NN over a billion vectors is infeasible. ANN trades a small recall hit for a 100x to 10,000x speedup. The reason vector search at scale exists.

Content-based filtering scores item features against a user profile. It handles item cold-start and often complements collaborative filtering.

How learned vector representations encode meaning, and why cosine similarity is the default metric for retrieval and recsys.

Linear models can't capture feature interactions. Polynomial models have too many parameters. Factorization machines find a middle path: factorize the interaction matrix and learn an embedding per feature.

Knowledge-graph embeddings turn link prediction into vector scoring. Compare TransE, DistMult, ComplEx, and RotatE by the relation patterns they can represent.

Ranking metrics are not differentiable training losses. Compare pointwise, pairwise, listwise, and lambda-weighted objectives by label quality, query structure, and target metric.

Decompose the user-item interaction matrix into user and item embeddings whose dot product approximates the rating. The original collaborative filtering.

Shared training can improve data efficiency or cause negative transfer. Diagnose task balance through labels, loss scales, gradients, calibration, and per-task outcomes.

Clicks reflect relevance and exposure. Use randomized data, propensities, IPS, self-normalization, or doubly robust estimates without hiding support and variance limits.

TF-IDF and BM25 are lexical retrieval baselines for search and RAG. BM25 adds term-frequency saturation and document-length normalization.

Encode queries and items with separate networks into a shared embedding space; retrieve by approximate nearest neighbors. The default architecture for industrial recommenders and search.

Measure whether visible reasoning contains useful evidence of behavior, while remembering that the trace may be incomplete, unfaithful, or changed by training.

Let new requests join an in-flight batch at every decode step instead of waiting for the slowest one. The other half of why vLLM is fast.

Same model, different samplers, very different outputs. The choice of decoder is often more impactful than the last percent of training. Know the tradeoffs.

I/O-aware exact attention replaces O(n²) HBM traffic with a tiled streaming softmax in SRAM. It is a core transformer kernel optimization.

Share K and V heads across query heads to shrink the KV cache 4-8x with negligible quality loss. Standard in modern decoder LLMs.

A core optimization in autoregressive decoding. Without it, generating 1000 tokens would require O(1000²) attention operations.

Approximate the softmax attention matrix with a low-rank or kernel factorization so cost is linear in sequence length.

Map assets, actors, trust boundaries, and attack paths across prompts, retrieval, tools, memory, generated code, and tenants.

What makes a 1M-token context model work. Position-encoding extension, attention kernels, KV-cache management, and the tradeoffs.

Use causal interventions, not attractive visualizations, to identify internal computations that produce model behavior.

Replace one large feed-forward block with N smaller experts and a router that activates only k of them per token. Trades parameter count for compute.

Create controlled models with known problematic behavior to test detection and mitigation, while keeping the induced behavior bounded and measurable.

Run h independent attention computations in parallel, then concatenate. Each head specializes in a different relation. The mechanism most senior candidates can write but few can motivate.

Treat the KV cache like virtual memory: allocate in fixed-size pages, share pages across sequences, eliminate fragmentation. The reason vLLM is the default LLM server.

Preference optimization is a measurement system: sampling policy, annotator protocol, disagreement, calibration, and shift determine the signal.

LLM inference has two cost regimes with very different bottlenecks. Mixing them up leads to wrong cost models and bad serving decisions.

Reduce model precision to shrink memory and speed up inference. The trade-offs are real but increasingly small with modern techniques.

The standard pattern for grounding LLMs in your own data. Reference page; the full essay is linked at the bottom.

How LLMs get from 'next-token predictor' to 'helpful assistant.' The post-training pipeline in 2026.

Modern LLMs usually replace sinusoidal positional encoding with RoPE, ALiBi, or related methods. Compare how they represent relative position and extrapolate.

The dominant position encoding for modern LLMs. Encodes relative position by rotating Q and K in 2D subspaces and supports several context-extension methods.

Oversight tries to judge work beyond unaided human ability; control tries to keep systems safe even when the model may be strategically untrusted.

Self-attention reads from one sequence; cross-attention reads from another. This input choice determines encoder-only, decoder-only, and encoder-decoder structures.

Replace the dense n×n attention mask with a sparse pattern that has O(n) non-zeros while preserving information flow across the full sequence.

Break the autoregressive serial bottleneck without changing the output distribution. 2-3× inference speedup, free.

Spend extra inference compute on several candidate solutions, search, tools, or revision, then select results with evidence.

The critical input layer between text and model. Tokenization mismatch is a frequent source of production LLM bugs.

Strip away the diagram clutter. A transformer is a stack of (residual + LayerNorm + (attention or FFN)) blocks. Understanding why each piece is there is more important than memorizing the diagram.

Trade compute for memory: drop activations during the forward pass and recompute them during the backward pass. The cheapest way to fit a larger model on the same GPU.

How Adam works, why AdamW is usually preferred, and which newer optimizers address memory, batch size, or curvature.

BatchNorm and LayerNorm normalize different axes and behave differently during training. The axis and train/eval behavior determine which one fits.

Accuracy isn't enough; you also want predictions to mean what they say. Calibration is the difference.

The pairing isn't arbitrary. Cross-entropy is the negative log-likelihood under a categorical distribution, and the softmax+CE gradient simplifies to (p − y), which is why it's stable.

Randomly zero out a fraction of activations during training. The simplest stochastic regularizer; still standard in vision and many NLP architectures.

Source, filter, deduplicate, decontaminate, mix, and audit data as a versioned model input rather than a pile of tokens.

Run several forward-backward passes before each optimizer step to simulate a larger effective batch size without the memory cost.

Cap the norm of the gradient before each optimizer step. The simplest and most reliable defense against training instability.

Replace one-hot targets with a softened distribution that puts ε mass on the wrong classes. Improves calibration, sometimes hurts retrieval.

Why almost every modern training run linearly warms up the LR over a few hundred steps and then decays it on a cosine to near zero.

A training spike is a first-bad-transition problem across data, numerics, optimizer, distributed state, software, and hardware.

A short cooldown applied to a mostly-trained checkpoint with a small fraction of candidate data mixed in. The standard mid-training probe for whether a new dataset is worth including.

How modern transformers train at 2-4× the throughput of FP32 without quality loss. The bit layouts matter; the loss-scaling recipe matters more.

Two data-augmentation schemes that train on convex combinations of pairs of inputs and their labels. Strong regularization for image classification; sometimes used in audio and tabular.

Use small training runs to estimate how loss changes with model size, data, and compute, then choose a training plan within a fixed budget.

The classical regularizers + the modern reality that SGD's noise is itself a regularizer. The hierarchy of choices when your model is overfitting.

Add a moving average of past gradients to the update. Smoother trajectories, faster convergence in narrow valleys, and the foundation of Adam's first moment.

Generate training examples for a clear capability, verify them with independent evidence, and protect diversity and held-out evaluation.

A systematic procedure for taking a model from 'not training' to 'tuned', the order of operations that separates disciplined practitioners from people who randomly change hyperparameters and hope.

L2 adds ½λ‖θ‖² to the loss; weight decay shrinks θ multiplicatively at each step. They are equivalent under SGD but not under Adam. Which is why AdamW exists.

Set the initial variance of each layer's weights so that activations and gradients neither explode nor vanish through depth. The single most impactful one-line decision in deep nets.

Warmup-Stable-Decay keeps the learning rate flat before a final decay. WSD-S adds single-path decay-and-return checkpoints when the final token budget is uncertain.

Z-loss penalizes the squared softmax log-partition value. It limits logit growth and can improve stability during long training runs or cooldowns.

Place each parallelism axis on hardware links that can carry its message size, frequency, and latency needs.

The communication primitives behind every distributed training job. All-reduce, all-gather, reduce-scatter, broadcast. What they do, costs, and when each is used.

Shard a long sequence across devices while preserving exact attention and controlling activation memory.

What happens when a rank fails, stalls, diverges, or corrupts data inside all-reduce, and which guarantees recovery can actually preserve.

How modern accelerators trade precision for speed. The bit layouts of every numeric format that appears in deep learning.

How modern training scales beyond a single GPU's memory by partitioning the optimizer state, gradients, and parameters across the data-parallel group.

Decide whether an accelerator operation is limited by compute or by data movement across HBM, caches, and on-chip memory.

Train a small student to match a large teacher's outputs. The student gets richer signal than from hard labels because the teacher's soft probabilities encode similarity structure.

Trace a model from raw data through transforms, features, labels, training, evaluation, and deployment. Version each contract for replay, rollback, deletion, and incident response.

Split the model across GPUs by layer; pipeline mini-batches through the stages. The way to scale across slow interconnects when TP isn't viable.

Use step traces, roofline limits, and communication timelines to find the exposed bottleneck in training and inference.

Set unimportant weights to zero, recover most of the accuracy. Unstructured pruning shrinks model size; structured pruning shrinks inference time. They solve different problems.

Concatenate multiple short examples into one fixed-length sequence to eliminate padding waste. The single largest throughput win for training on skewed-length corpora.

Predict which collective communication a distributed matrix multiplication needs from its sharded axes.

Choose data, tensor, pipeline, and sharded parallelism by checking memory, communication, topology, and scaling efficiency.

Split a single matrix multiplication across multiple GPUs. The way to fit one transformer layer that doesn't fit on a single device.

Estimate parameters, training FLOPs, activation memory, and KV-cache memory from a small set of model dimensions.

The framework for proving a model change actually helps. Statistical power, novelty effects, network effects, all the things people get wrong.

The 2x2 (or KxK) table of predictions vs. truth that every classification metric is computed from. The Rosetta stone of binary classification.

Hold-out, k-fold, stratified, grouped, and time-series CV. And when each one is and isn't appropriate.

A feature is valid only if it was available when the prediction was made. Prevent target, temporal, group, preprocessing, and evaluation leakage.

Choose actions from calibrated probabilities, error costs, and capacity constraints. Use separate thresholds for automatic action, human review, and abstention.

Production policies control which labels arrive and when. Track label maturity, selection, action propensities, and feedback before training or evaluating a model.

Check that an evaluation measures the claimed ability, uses independent data, and still predicts performance on the target task.

How well do predicted probabilities match empirical frequencies? Bin predictions by confidence, compare bin-mean confidence to bin-accuracy.

Use a model to score open-ended outputs only after testing its rubric, bias, agreement, and resistance to manipulation.

How to explain a model's predictions: the split between intrinsic and post-hoc methods, global vs local, and the four techniques interviewers expect (feature importance, SHAP, LIME, and saliency / Grad-CAM).

The standard intrinsic metric for language models. What it measures, what units to use, and why it's a poor end-product evaluation.

The three metrics every classifier interview asks about. Their definitions, when to optimize which, and the F-beta generalization.

Beyond binary precision-recall: how to measure ranking quality when order matters and labels are graded.

Compare models with matched resources, repeated runs, complete records, and tests that another person can reproduce.

What ROC-AUC and PR-AUC measure, when to use which, and why ROC-AUC is misleading on heavy class imbalance.

Derive Chinchilla limits for dense and MoE models, implement an MoE layer in PyTorch, and fuse projections in Triton when F exceeds D.