ARIMA Models | STAT-101

The Workhorse of Time Series Forecasting

ARIMA — Autoregressive Integrated Moving Average — is the most widely used class of models for time series forecasting. It unifies three ideas: autoregression (predicting from past values), differencing (achieving stationarity), and moving averages (modeling past forecast errors). Together, these three components handle a wide variety of real-world time series with trend and autocorrelation.

The Box–Jenkins methodology, developed in the early 1970s, formalized how to identify, estimate, and validate ARIMA models using ACF and PACF diagnostics. Despite the advent of machine learning, ARIMA and its variants remain competitive benchmarks for many forecasting tasks, especially with shorter series or when interpretability matters.

ARIMA in Three Letters

AR(p) — Autoregressive: current value depends on p past values.

I(d) — Integrated: d rounds of differencing to achieve stationarity.

MA(q) — Moving Average: current value depends on q past forecast errors.

ARIMA(p, d, q) = AR(p) + d differences + MA(q)

The Autoregressive Model AR(p)

An autoregressive model of order p predicts the current value as a linear combination of its p most recent past values plus white noise. The idea is that a process has memory: today's value is influenced by what happened yesterday, two days ago, up to p days ago.

Autoregressive Model AR(p)

X_t = \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \varepsilon_t

X_t is predicted from its p lagged values. The φ_i are the autoregressive coefficients estimated from data. ε_t is white noise.

The AR(1) model — the simplest case — says that today's value is φ times yesterday's value plus noise. If |φ| < 1, the series is stationary. If φ = 1, we have a random walk, which is non-stationary. If |φ| > 1, the series explodes. Stationary AR processes have an ACF that decays geometrically and a PACF that cuts off sharply after lag p.

The Moving Average Model MA(q)

A moving average model of order q models the current value as a linear combination of the current and q past white noise shocks (forecast errors). Unlike the AR model, the MA model has finite memory: the impact of a shock dies out exactly after q periods.

Moving Average Model MA(q)

X_t = \varepsilon_t - \theta_1 \varepsilon_{t-1} - \theta_2 \varepsilon_{t-2} - \cdots - \theta_q \varepsilon_{t-q}

The current value depends on the current shock ε_t and the q previous shocks. The θ_i are the moving average coefficients.

MA processes are always stationary for any finite q. Their ACF cuts off sharply after lag q — a distinctive signature that makes them easy to identify in practice. Their PACF, however, decays gradually.

ARMA(p, q): Combining Both

The ARMA(p, q) model combines autoregression and moving averages. Many real-world time series are more compactly represented by mixing a small number of AR and MA terms than by using a pure AR or MA model of higher order. The principle of parsimony — preferring simpler models — favors ARMA over high-order pure models.

ARMA(p, q) Model

X_t = \sum_{i=1}^{p} \phi_i X_{t-i} + \varepsilon_t - \sum_{j=1}^{q} \theta_j \varepsilon_{t-j}

The ARMA model combines p autoregressive terms and q moving average terms. For ARMA, both the ACF and PACF decay gradually rather than cutting off cleanly.

Differencing and the Integrated Component

Real-world series often have trends and are therefore non-stationary. ARMA models require stationarity. The solution is differencing: subtract each observation from the one before it. One round of differencing removes a linear trend; two rounds remove a quadratic trend. The number of differencing operations needed is the d in ARIMA(p, d, q).

First Difference (d=1)

\nabla Y_t = Y_t - Y_{t-1}

Removes a linear trend. Stock returns are often computed this way.

Second Difference (d=2)

\nabla^2 Y_t = \nabla Y_t - \nabla Y_{t-1}

Removes a quadratic trend. Rarely needed beyond d=2 in practice.

The Augmented Dickey–Fuller (ADF) test is the standard statistical test for non-stationarity. A significant ADF result (p-value < 0.05) suggests the series is stationary; a non-significant result indicates differencing is needed. After differencing, retest until stationarity is confirmed.

ARIMA(p, d, q)

The full ARIMA model applies an ARMA(p, q) to the d-times-differenced series. In practice, d is almost always 0, 1, or 2; most real series need only one round of differencing.

Common ARIMA Specifications

ARIMA(1,1,0): AR(1) on the differenced series — a simple, robust choice for many economic series.

ARIMA(0,1,1): MA(1) on the differenced series — equivalent to Simple Exponential Smoothing.

ARIMA(0,2,2): MA(2) on the twice-differenced series — equivalent to Holt's linear trend method.

ARIMA(1,0,0): A stationary AR(1) process — appropriate when d=0 and PACF cuts off after lag 1.

Model Identification: The Box–Jenkins Approach

Box and Jenkins proposed a systematic three-stage cycle for fitting ARIMA models: identification, estimation, and diagnostic checking.

Identification: Use ACF and PACF plots of the (differenced) series to determine p and q. A PACF that cuts off after lag p with a gradual ACF suggests AR(p). An ACF that cuts off after lag q with a gradual PACF suggests MA(q). Both decaying gradually suggests ARMA. Use the ADF test to determine d.

Estimation: Fit the model by maximum likelihood estimation. Modern software (Python’s statsmodels, R’s forecast) does this automatically. Compare competing models with information criteria — AIC and BIC both penalize model complexity, helping you avoid overfitting.

Akaike Information Criterion

\text{AIC} = 2k - 2\ln(\hat{L})

k = number of parameters; lower AIC = better balance of fit and complexity.

Bayesian Information Criterion

\text{BIC} = k\ln(n) - 2\ln(\hat{L})

n = sample size; BIC penalizes complexity more heavily than AIC for large n.

Diagnostic checking: Examine the residuals of the fitted model. They should look like white noise: no autocorrelation (check with the Ljung–Box test), roughly normal, and constant variance. If patterns remain, the model is mis-specified and you should iterate.

Seasonal ARIMA (SARIMA)

Many real series have seasonal patterns — monthly, quarterly, or weekly cycles. The SARIMA(p, d, q)(P, D, Q)_m model extends ARIMA with a seasonal component operating at period m. The uppercase (P, D, Q) describe the seasonal AR order, seasonal differencing, and seasonal MA order; the lowercase (p, d, q) describe the non-seasonal part.

SARIMA Notation

SARIMA(1,1,1)(1,1,1)₁₂ means: non-seasonal AR(1), one difference, MA(1), plus seasonal AR(1), one seasonal difference, seasonal MA(1), all at period m=12 (monthly data).

Seasonal differencing removes annual seasonality: ∇₁₂Y_t = Y_t − Y_t−12.

Identifying a SARIMA model follows the same logic as ARIMA, but you now look at the ACF and PACF at the seasonal lags (multiples of m) in addition to the non-seasonal lags. Seasonal spikes in the ACF at lags 12, 24, 36 on monthly data are a clear sign that a seasonal component is needed.

In the next lesson, we turn from model fitting to the practical art of forecasting: generating point and interval predictions, measuring accuracy with MAE, RMSE and MAPE, and comparing ARIMA against exponential smoothing methods and modern alternatives like Prophet.

Key Takeaways

ARIMA(p, d, q) combines p autoregressive terms, d rounds of differencing, and q moving average terms into a single unified model.
AR processes have ACFs that decay gradually and PACFs that cut off after lag p; MA processes have the opposite signature.
Differencing (d) is applied until the ADF test confirms stationarity; d=1 is sufficient for most real-world series.
The Box–Jenkins cycle — identify via ACF/PACF, estimate by MLE, check residuals for white noise — is the standard workflow.
AIC and BIC balance goodness-of-fit against model complexity; always prefer the simpler model that explains the data adequately.
SARIMA adds a seasonal layer (P, D, Q)_m to handle periodic patterns such as monthly or quarterly cycles.