Simple: At a loud dinner party you do not hear every voice equally — you tune in to the two people whose words actually bear on what you are about to say. Every word in a sentence does that, to every other word, at the same moment.

Technical: Scaled dot-product attention scores every query against every key, softmaxes those scores into weights, and returns the weighted sum of the values. It is an all-pairs operation with no recurrence, which is why the whole sequence resolves in one parallel step instead of one timestep at a time (Vaswani et al., 2017).

Simple: Nobody redesigned the engine between 2018 and 2023. They kept the same engine and built it bigger, on more fuel — and past a certain size it started doing tricks nobody had taught it, like following an instruction it had only seen described.

Technical: The GPT line holds the decoder-only, next-token-prediction objective fixed across four generations and scales parameters, data and compute. GPT-3 (2020) is the paper that named the payoff: in-context learning, where few-shot examples in the prompt substitute for gradient updates — a capability the training objective never optimised for.

Simple: A writer composing a sentence can only use the words already on the page. A proofreader reading the finished page can use both sides of any gap — which is why proofreading catches things writing cannot. GPT writes; BERT proofreads.

Technical: BERT is an encoder-only transformer with no causal mask, so every position attends to the full sequence in both directions. That makes autoregressive generation impossible and representation quality excellent, which is why the family survives in classification, ranking and retrieval rather than in chat.

Simple: Sticking the note on is cheap. The reason it works is separate and less obvious: the correction a new job needs is usually simple — a nudge in a few consistent directions, not a rewrite of every page. A short note can hold a simple correction. It could not hold a complicated one.

Technical: LoRA freezes W and learns ΔW = A·B with A of shape d×r and B of shape r×d, so ΔW is constrained to rank at most r. The bet is that the weight update fine-tuning wants has low intrinsic rank. Parameters trained per adapted matrix are 2·d·r, linear in r — and at inference B·A merges back into W, so latency is unchanged (Hu et al., 2021).

Simple: There is more than one place to clip the note. You can write in the margin next to the text (a small module inserted between layers), or you can staple a covering memo to the front so everything after it is read differently (a learned preamble). LoRA is one of these options, not the whole family.

Technical: PEFT is the umbrella for methods that adapt a frozen backbone by training a small set of new parameters. Adapters insert bottleneck MLP modules inside each block; prefix tuning prepends learned key/value vectors to every attention layer; LoRA reparameterises the weight update itself. They differ in where the trainable parameters sit and in whether they add inference cost — adapters and prefixes do, merged LoRA does not.

Simple: A child raised on a handful of photos still knows a cat is a cat upside down, or in the corner of the frame — that assumption comes built in. ViT is not born with it. Show it few pictures and it flounders; show it enormously many and it works the rule out itself, then applies it better than the built-in version.

Technical: A CNN hard-codes translation equivariance and locality as architectural priors. ViT removes them: patches are embedded, position is learned, and attention is global from layer one. With less data that missing prior costs it accuracy against a CNN; pre-trained on a very large corpus it overtakes, which is the scaling result the paper reports (Dosovitskiy et al., 2020).

Simple: The sketcher is graded on how closely the redrawing matches the original, so when it is unsure it hedges — and a hedged line is a soft line. The forger is graded only on whether the inspector is fooled, so hedging earns nothing. Each one’s weakness is the direct cost of what it is being graded on.

Technical: A VAE maximises a reconstruction likelihood plus a KL term pulling the latent posterior towards a prior; the pixel-wise reconstruction loss is what averages away high-frequency detail, and the KL term is what buys the continuous latent space you can interpolate through. A GAN optimises a minimax objective against a discriminator with no reconstruction term at all — hence sharpness, and hence mode collapse and unstable training as the standing failure modes.

Simple: Un-blurring in one heroic go is a job for an artist. Un-blurring by a hair, over and over, is a job for a technician — and a technician you can actually train, because you can manufacture unlimited practice material for free just by blurring photographs you already have.

Technical: The forward process is a fixed Gaussian noising schedule with no learned parameters, so training pairs are generated for nothing and any timestep can be sampled directly. The network learns only to predict the noise added at a given step; sampling subtracts that prediction iteratively. Latent diffusion runs the loop in a compressed autoencoder space rather than pixel space, which is the change that brought the cost down to consumer hardware (Ho et al., 2020; Rombach et al., 2022).

Simple: An open-book exam beats a closed-book one, and not because the student got cleverer. The student can now cite the page. Closed-book, a confident student who has forgotten the fact will simply invent one that sounds right — and sound exactly as confident doing it.

Technical: RAG splits knowledge from parameters: a retriever fetches passages at query time and the generator conditions on them, so updating the corpus updates the answers with no retraining. It narrows the hallucination surface rather than closing it — a model can still misread, over-generalise from, or ignore a retrieved passage, and a wrong retrieval is a wrong answer with a citation attached (Lewis et al., 2020).