STAT 101
M05 · L03
Module 5 · Lesson 3

Bootstrap Methods

What if you don’t know the population distribution? Pull yourself up by your own data. Resample from what you have and let computation do the math.

01 / 12
STAT 101
M05 · L03
Bradley Efron, 1979

The Audacious Idea

Classical inference needs a formula for every statistic. Bootstrap’s insight: treat the sample as a stand-in for the population. Simulate the sampling process using data you already have.

The substitution
Unknown F → Empirical F̂ₙ
Derive → Simulate B times
02 / 12
STAT 101
M05 · L03
Four simple steps

The Algorithm

  • Step 1: Draw n observations with replacement from your data → bootstrap sample x*
  • Step 2: Compute the statistic T* = T(x*)
  • Step 3: Repeat B times (B = 1,000–10,000)
  • Step 4: Analyze the distribution of {T*₁, …, T*_B}
03 / 12
STAT 101
M05 · L03
No formula derivation needed

Bootstrap Standard Error

Bootstrap SE
\widehat{\mathrm{SE}}_{\mathrm{boot}} = \sqrt{\dfrac{1}{B-1}\sum_{b=1}^{B}(T^*_b - \bar{T}^*)^2}

Standard deviation of the B bootstrap statistics. Works for any estimator — mean, median, correlation, ratio.

04 / 12
STAT 101
M05 · L03
Simplest bootstrap CI

Percentile Interval

95% Percentile CI
\bigl[\,T^*_{(\alpha/2)},\; T^*_{(1-\alpha/2)}\,\bigr]

Take the 2.5th and 97.5th percentiles of the bootstrap distribution. Intuitive and widely applicable when n is large.

05 / 12
STAT 101
M05 · L03
Bias-corrected version

Basic Bootstrap CI

Pivot / Reflection CI
\bigl[\,2\hat{T} - T^*_{(1-\alpha/2)},\; 2\hat{T} - T^*_{(\alpha/2)}\,\bigr]
Gold standard
BCa (bias-corrected & accelerated) — adjusts for bias AND skewness. Best coverage for small n.
06 / 12
STAT 101
M05 · L03
The power of universality

Works for Any Statistic

  • Median: no closed-form SE exists — bootstrap provides it
  • Correlation: more robust than Fisher z-transform
  • Ratio of means: no parametric formula
  • AUC / F1-score: standard for ML metrics
  • Regression coefficients: robust to non-normal residuals
07 / 12
STAT 101
M05 · L03
Two flavors

Parametric vs. Nonparametric

Nonparametric
Resample the data. No assumption. Universal.
Parametric
Simulate from fitted model. Efficient if model correct.
08 / 12
STAT 101
M05 · L03
Theoretical foundation

Why It Works

  • F̂ₙ converges to F (Glivenko–Cantelli theorem)
  • Bootstrap distribution of T*−T̂ converges to the sampling distribution of T̂−θ
  • Bootstrap CIs are asymptotically valid
  • Quality depends on how well F̂ₙ approximates F
  • Larger n → better approximation
09 / 12
STAT 101
M05 · L03
Known failure modes

When Bootstrap Fails

  • Sample max/min: bootstrap max can’t exceed observed max
  • Heavy tails: infinite variance → bootstrap may not converge
  • Tiny samples (n < 15): F̂ₙ too sparse; poor coverage
  • Dependent data: use block bootstrap for time series
  • Model selection: LASSO variable selection is not bootstrap-stable
10 / 12
STAT 101
Knowledge Check

Check what stuck

Four questions from this lesson. Answer to see why — the explanation appears whether you were right or wrong. Nothing is scored or saved.

Question 1 of 0
Score 0/0

11 / 12
STAT 101
Summary
Recap

What you learned

  • Resample with replacement B times; compute the statistic each time
  • Bootstrap SE = std dev of the B replicates — no formula needed
  • Percentile CI = [T*_(α/2), T*_(1−α/2)] — simplest method
  • BCa CI — best coverage when bootstrap is skewed or biased
  • Nonparametric: resample data. Parametric: simulate from fitted model
  • Fails for extremes, heavy tails, small n, and dependent data
Module 5 complete
Up next — Module 6: Hypothesis Testing →
Next lesson All lessons Read article
12 / 12