STAT 101
M08 · L03
Module 8 · Lesson 3
Non-Parametric Regression
Drop the assumed formula. Let the data draw its own curve under a smoothness constraint — for when you don’t know the functional form.
01 / 07
STAT 101
M08 · L03
The smooth histogram
Kernel Density Estimation
KDE
\hat{f}(x) = \tfrac{1}{nh}\sum K\!\left(\tfrac{x - x_i}{h}\right)
A bump over each point; the bandwidth h sets the smoothness (Module 2, Lesson 1). The same local-averaging idea, applied to y, is non-parametric regression.
02 / 07
STAT 101
M08 · L03
Fit one point at a time
LOESS: Local Regression
Local weighted fit
\min_{a,b} \sum w_i(x)(y_i - a - b x_i)^2
A tiny weighted regression at each x, weighting nearby points more. The span is the smoothness knob.
03 / 07
STAT 101
M08 · L03
Stitched polynomials
Splines
Low-degree polynomials between knots, joined smoothly. A smoothing spline adds a roughness penalty — a cousin of Ridge — and powers generalized additive models.
04 / 07
STAT 101
M08 · L03
The Bayesian member
Gaussian Processes
A prior over functions → a posterior over all curves that fit. You get honest uncertainty bands that widen where data is sparse — a bridge to Module 9.
05 / 07
STAT 101
Summary
Recap
What you learned
- Non-parametric regression assumes no formula — only smoothness
- KDE, LOESS, splines, Gaussian processes — four routes to a flexible curve
- Each has one smoothness knob = the bias-variance trade-off, set by CV
- Gaussian processes add honest, data-dependent uncertainty bands
Up Next — Module 9
07 / 07