Regression Models over Time: ARIMA and Time Series

13/07/2026, 업데이트 29/10/2026 •0•0

Understand the AR, I and MA components, ARIMA(p,d,q) notation, a step-by-step forecast, statsmodels code and exercises with an answer key.

작성자:Leon

번역자Editorial

개정자Editorial

ARIMA (AutoRegressive Integrated Moving Average) is a statistical model widely used to analyse and forecast time series. It combines three components:

AR (autoregressive): dependence on the series' own past values.
I (integrated): differencing to make the series stationary.
MA (moving average): dependence on past errors.

This gives ARIMA a framework far better suited to time series than a simple linear regression against time. We will explain what ARIMA is and then compare it with that regression to understand why it is usually better when we deal with data over time.

Components of the ARIMA Model

1. AR(p) – Autoregressive of Order p

Autoregressive series are those in which today's value depends on the series' own past values. Think of monthly inflation: a month of high inflation tends to be followed by another month of high inflation, because contracts and prices are adjusted based on the past. The same holds for daily energy demand, or for a day's average temperature relative to the previous day.

The AR(p) model writes \( Y_t \) as a linear combination of its past values:

\[ Y_t = c + \phi_1 Y_{t-1} + \phi_2 Y_{t-2} + \dots + \phi_p Y_{t-p} + \epsilon_t \]

where p is the number of lags used. To choose p, we look at the partial autocorrelation function (PACF): in an AR(p) it cuts off after lag p.

2. I(d) – Integration of Order d

d is the number of differencing steps needed to make the series stationary, that is, without an upward or downward trend.

A time series is stationary when its statistical properties (mean, variance and autocorrelation) do not change over time. This is essential for ARIMA, which assumes stationarity (after differencing) to produce reliable forecasts. The formal test is the Augmented Dickey-Fuller (ADF) test, which we will see later; for now, we will judge stationarity by looking at the series.

Look at the following series and notice that, by working with the difference between consecutive values, the series now fluctuates around a constant mean:

Month (t)	Sales (Yₜ)	1st difference (∇Yₜ)
1	100	–
2	120	120 − 100 = 20
3	130	130 − 120 = 10
4	150	150 − 130 = 20
5	170	170 − 150 = 20
6	190	190 − 170 = 20

So we apply I(d) when we notice an upward or downward trend in the series. If the first difference still has a trend (for example, when the series grows quadratically), we use I(2), that is, we difference again until the mean is constant.

3. MA(q) – Moving Average of Order q

We use MA(q) when the series suffers shocks that affect only a few subsequent periods and then fade: the series has a "short memory" of its errors. In an MA(1), today's value is affected by yesterday's error; in an MA(2), by the errors of yesterday and the day before. Examples of data with temporary impacts: stock or product prices after a one-off event, demand after a flash sale.

The MA(q) model writes \( Y_t \) as a linear combination of past errors:

\[ Y_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + \dots + \theta_q \epsilon_{t-q} \]

q: number of error terms used.
ACF (autocorrelation function): identifies the order q, because in an MA(q) it cuts off after lag q.

Example of an MA(2) with \( \mu = 50 \), \( \theta_1 = 0.8 \) and \( \theta_2 = 0.3 \), assuming errors equal to zero before day 1:

Day (t)	Error (εₜ)	MA(2) model calculation	Price (Yₜ)
1	+2.00	Y₁ = 50 + 2.00 + 0.8×0 + 0.3×0	52.00
2	−1.00	Y₂ = 50 + (−1.00) + 0.8×2 + 0.3×0	50.60
3	+1.50	Y₃ = 50 + 1.50 + 0.8×(−1) + 0.3×2	51.30
4	−0.80	Y₄ = 50 + (−0.80) + 0.8×1.5 + 0.3×(−1)	50.10
5	+0.34	Y₅ = 50 + 0.34 + 0.8×(−0.8) + 0.3×1.5	50.15
6	−0.28	Y₆ = 50 + (−0.28) + 0.8×0.34 + 0.3×(−0.8)	49.75

Notice that the price always returns to around 50: each shock only influences the following two days.

A series takes the values 10, 14, 18, 22, 26 in months 1 to 5. What is the most appropriate order of integration d?

Answer: d = 1 — the first difference gives 4, 4, 4, 4, a series with a constant mean, so one differencing step is enough. d = 2 would be needed if the first difference were still growing, as in a quadratic series.

In an MA(1) with μ = 100 and θ1 = 0.5, the error in the last period was +4. What is the forecast for the next period (the future error has an expected value of zero)?

Answer: 102 — the forecast is μ plus θ1 times the last error: 100 + 0.5 × 4 = 102. The 104 option adds the whole error, forgetting to multiply by θ1.

ARIMA(p, d, q) Notation

For example, in an ARIMA(1, 1, 1):

AR(1): the differenced series depends on its previous value;
I(1): one differencing step is applied;
MA(1): it also depends on the previous error, \( \epsilon_{t-1} \).

ARIMA(1,1,1) equation:

\[ \nabla Y_t = c + \phi_1 \nabla Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \]

Practical Example: Forecasting Monthly Sales

Step 1: Load and Visualize the Data

Monthly sales data (in thousands):

Month	Sales
1	100
2	120
3	130
4	150
5	170
6	190

Step 2: Check Stationarity

The series has a clear upward trend, so it is not stationary. The ADF test confirms it (running statsmodels' adfuller on these 6 points gives a p-value of about 0.99, far from 0.05). We then apply the first difference:

\[ \nabla Y_t = Y_t - Y_{t-1} \]

Month	Sales	\( \nabla Y_t \)
2	120	20
3	130	10
4	150	20
5	170	20
6	190	20

The differenced series fluctuates around a constant mean. Note: with so few observations, the ADF test has very little power, and in practice it needs dozens of points; here the decision is visual.

Step 3: Define the ARIMA(1,1,1) Model

To keep the calculations simple, we will use the model without a constant:

\[ \nabla Y_t = \phi_1 \nabla Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \]

Where:

\( \nabla Y_t = Y_t - Y_{t-1} \) (differenced series);
\( \phi_1 \) = AR(1) coefficient;
\( \theta_1 \) = MA(1) coefficient;
\( \epsilon_t \) = error at time \( t \).

Step 4: Choose the Coefficients (AR and MA)

Since we are doing the calculations by hand, we will use typical values for illustration:

\( \phi_1 = 0.6 \) (AR)
\( \theta_1 = -0.3 \) (MA)

Packages such as Python's statsmodels estimate these coefficients by maximum likelihood: they write the model in state-space form and use the Kalman filter to find the values that make the observed data most probable (which, with normal errors, is roughly equivalent to minimizing the sum of squared errors).

Step 5: Set Up the Forecasts

Each month's forecast uses the actual difference of the previous month and the previous month's error:

Month	∇Yₜ (actual)	ARIMA(1,1,1) forecast
3	10	\( \widehat{\nabla Y_3} = 0.6 \times 20 + (-0.3) \times \epsilon_2 \)
4	20	\( \widehat{\nabla Y_4} = 0.6 \times 10 + (-0.3) \times \epsilon_3 \)
5	20	\( \widehat{\nabla Y_5} = 0.6 \times 20 + (-0.3) \times \epsilon_4 \)
6	20	\( \widehat{\nabla Y_6} = 0.6 \times 20 + (-0.3) \times \epsilon_5 \)

Step 6: Calculate the Errors (\( \epsilon_t \))

In month 2 there is no previous difference to use yet. Since the model has no constant, the forecast without information is 0, and the initial error is the difference itself: \( \epsilon_2 = 20 - 0 = 20 \).

Month	∇Yₜ (actual)	Forecast (\( \widehat{\nabla Y_t} \))	Error (\( \epsilon_t = \nabla Y_t - \widehat{\nabla Y_t} \))
2	20	0 (no history)	\( \epsilon_2 = 20 - 0 = 20 \)
3	10	\( 0.6 \times 20 + (-0.3) \times 20 = 6 \)	\( \epsilon_3 = 10 - 6 = 4 \)
4	20	\( 0.6 \times 10 + (-0.3) \times 4 = 4.8 \)	\( \epsilon_4 = 20 - 4.8 = 15.2 \)
5	20	\( 0.6 \times 20 + (-0.3) \times 15.2 = 7.44 \)	\( \epsilon_5 = 20 - 7.44 = 12.56 \)
6	20	\( 0.6 \times 20 + (-0.3) \times 12.56 = 8.23 \)	\( \epsilon_6 = 20 - 8.23 = 11.77 \)

Step 7: Forecast the Next Month (Month 7)

Using the latest values:

\[ \widehat{\nabla Y_7} = 0.6 \times 20 + (-0.3) \times 11.77 = 8.47 \]

\[ \hat{Y}_7 = Y_6 + \widehat{\nabla Y_7} = 190 + 8.47 = 198.47 \]

ARIMA(1,1,1) model summary

Equation used: \( \nabla Y_t = 0.6 \nabla Y_{t-1} - 0.3 \epsilon_{t-1} + \epsilon_t \)
Forecast for month 7: about 198.5

Notice that all the errors were positive: the model underestimated growth in every month. This happens because sales rise by 18 per month on average, while the model without a constant assumes that the difference fluctuates around zero. For a trending series, the ARIMA(1,1,1) should include the constant c (called the drift), which represents this average growth.

Doing the Same in Python

In current statsmodels, ARIMA lives in statsmodels.tsa.arima.model. With d = 1, the growth constant is requested with trend="t":

import pandas as pd
from statsmodels.tsa.arima.model import ARIMA

vendas = pd.Series([100, 120, 130, 150, 170, 190], index=pd.period_range("2024-01", periods=6, freq="M"))
modelo = ARIMA(vendas, order=(1, 1, 1), trend="t")
resultado = modelo.fit()
print(resultado.params)
print(resultado.forecast(steps=1))

The estimated drift is close to 17.8 per month and the forecast for month 7 comes out at around 206, above the 198.5 of the model without a constant. With only 6 observations, however, the estimated AR and MA coefficients are very unstable: use the code to learn the syntax, not to trust these numbers.

In an ARIMA(1,1,0) without a constant, with φ1 = 0.5, the last two values of the series were 190 and 200. What is the forecast for the next period?

Answer: 205 — the last difference is 200 minus 190 = 10; the forecast difference is 0.5 × 10 = 5; adding it to the last value, 200 + 5 = 205. The 210 option repeats the whole last difference, as in a random walk with drift, ignoring φ1.

In the sales example, all the errors of the ARIMA(1,1,1) without a constant were positive. What does this indicate?

Answer: That the constant (drift) is missing: the series grows 18 per month on average and the model assumes zero average growth — errors that always have the same sign are a sign of bias. Changing the sign of θ1 would not fix it, because the problem lies in the average level of the differenced series.

Differences Between the ARIMA Models

Model	Formula	When to use?	Practical example
ARIMA(0,0,0) (white noise)	\( Y_t = \mu + \epsilon_t \)	When the series is pure noise, with no patterns	Temperature of an air-conditioned room
ARIMA(0,1,0) (random walk)	\( Y_t = Y_{t-1} + \epsilon_t \)	Series with a stochastic trend	Stock prices
ARIMA(1,0,0) (AR(1))	\( Y_t = c + \phi_1 Y_{t-1} + \epsilon_t \)	When Yₜ depends only on Yₜ₋₁	Daily energy demand
ARIMA(0,0,1) (MA(1))	\( Y_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} \)	Temporary shocks affecting only 1 period	Sales after a promotion
ARIMA(1,0,1) (ARMA(1,1))	\( Y_t = c + \phi_1 Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \)	When there is memory of the past value and of the past error	Commodity prices
ARIMA(1,1,0)	\( \nabla Y_t = \phi_1 \nabla Y_{t-1} + \epsilon_t \)	Trend + autocorrelation at lag 1	GDP growth
ARIMA(0,1,1)	\( \nabla Y_t = \theta_1 \epsilon_{t-1} + \epsilon_t \)	Trend + temporary shocks	Oil prices
ARIMA(1,1,1)	\( \nabla Y_t = \phi_1 \nabla Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \)	Combines trend, memory of the value and memory of the error	Sales with trend and shocks

None of these models captures seasonality (peaks that repeat every year, such as Christmas sales). For that there is the SARIMA extension, which adds seasonal AR, I and MA terms.

A store's monthly sales grow year after year and peak every December. Which model is the most suitable?

Answer: SARIMA, with differencing and seasonal terms of period 12 — the trend calls for differencing and the annual peak calls for the seasonal part. A plain ARIMA or a straight line against time cannot reproduce the December peak.

Why Use ARIMA Instead of Linear Regression Against Time?

The choice between ARIMA and linear regression against time depends on the structure of the data and on the statistical assumptions. Linear regression against time (simple model) assumes the series can be written as:

\[ Y_t = \beta_0 + \beta_1 \cdot t + \epsilon_t \]

Where:

\( t \) = time variable (e.g. 1, 2, 3, ...);
\( \epsilon_t \) = random error.

Its advantage is that it is simple to implement and interpret, and it works well when there is a clear linear trend (constant growth). But it has limitations:

It ignores autocorrelation: in time series, past values influence future ones (product demand, stock prices), and the regression treats each observation as independent, which is rarely true.
It does not handle seasonality or cycles if t is the only regressor: repeating patterns (such as Christmas sales) stay in the residuals.
It requires uncorrelated residuals: in time series, residuals are usually autocorrelated (if demand was high today, it may be high tomorrow too), which makes the regression's t-tests and confidence intervals misleading.

Direct Comparison: Linear Regression vs. ARIMA

Criterion	Linear regression against time	ARIMA
Trend	Models only a linear trend	Removes the trend through differencing
Autocorrelation	Ignores it	Captures it (AR and MA terms)
Seasonality	Does not handle it (only with extra variables)	Only in the SARIMA extension
Residuals	Assumes independence	Models the dependence; the final residuals should be white noise
Complexity	Low	High (choice of p, d, q)

Why does a simple linear regression of Y against t usually give misleading t-tests for time series?

Answer: Because the residuals are usually autocorrelated, violating the independence assumption — with dependent residuals, the standard errors are underestimated and the tests flag too much significance. Contrary to the last option, regressions against time often have a high R² precisely because of the trend.

Exercises

In the exercises with an MA term, when there is no previous value to use, the forecast is just the constant c (as in Step 6 of the example, where c = 0).

1. What is the difference between the AR and MA components in an ARIMA?

a) AR uses past values; MA uses past errors.
b) AR is for trend; MA is for seasonality.
c) Both are the same.

2. Forecast month 7 using an ARIMA(0,1,0) (random walk, without a constant).

Month	Sales
1	100
2	120
3	130
4	150
5	170
6	190

3. Fit an ARIMA(1,0,0) to the series below and forecast t = 5 with \( \phi_1 = 0.6 \), \( \theta_1 = -0.3 \) and \( c = 10 \). (Note: one of the given coefficients is not used in this model.)

t	Y
1	10
2	15
3	20
4	25

4. Fit an ARIMA(1,0,1) to the series below and forecast t = 5 with \( \phi_1 = 0.6 \), \( \theta_1 = -0.3 \) and \( c = 10 \).

t	Y
1	10
2	15
3	20
4	25

5. Fit an ARIMA(1,1,1) to the series below and forecast t = 5 with \( \phi_1 = 0.6 \), \( \theta_1 = -0.3 \) and \( c = 10 \).

t	Y
1	10
2	15
3	20
4	25

6. Use an ARIMA(1,1,0) to compute the forecast for every possible time, from 2 to 6 (6 is the forecast for the period after the data). Use \( \phi_1 = 0.6 \) and \( c = 20 \) (the \( \theta_1 = 0.3 \) does not enter this model). Then calculate the SEE of the forecasts for times 2 to 5.

Time	Y (sales)
1	80
2	70
3	90
4	110
5	120

7. Use an ARIMA(1,1,1) to compute the forecast for every possible time, from 2 to 6. Use \( \phi_1 = 0.6 \), \( \theta_1 = 0.3 \) and \( c = 20 \).

Time	Y (sales)
1	80
2	70
3	90
4	110
5	120

Answer Key

1. a)

2. In a random walk, the forecast is the last value: \( \hat{Y}_7 = Y_6 = 190 \).

3. \( \hat{Y}_5 = 10 + 0.6 \times 25 = 25 \). \( \theta_1 \) does not enter, because the ARIMA(1,0,0) has no MA term.

4. Forecasts and errors: \( \hat{Y}_1 = 10 \) (just the constant), \( \epsilon_1 = 0 \); \( \hat{Y}_2 = 10 + 0.6 \times 10 = 16 \), \( \epsilon_2 = -1 \); \( \hat{Y}_3 = 10 + 0.6 \times 15 - 0.3 \times (-1) = 19.3 \), \( \epsilon_3 = 0.7 \); \( \hat{Y}_4 = 10 + 0.6 \times 20 - 0.3 \times 0.7 = 21.79 \), \( \epsilon_4 = 3.21 \). Forecast: \( \hat{Y}_5 = 10 + 0.6 \times 25 - 0.3 \times 3.21 \approx 24.04 \).

5. Differences: 5, 5, 5 (t = 2, 3, 4). \( \widehat{\nabla Y_2} = 10 \), \( \epsilon_2 = -5 \); \( \widehat{\nabla Y_3} = 10 + 0.6 \times 5 - 0.3 \times (-5) = 14.5 \), \( \epsilon_3 = -9.5 \); \( \widehat{\nabla Y_4} = 10 + 3 - 0.3 \times (-9.5) = 15.85 \), \( \epsilon_4 = -10.85 \); \( \widehat{\nabla Y_5} = 10 + 3 - 0.3 \times (-10.85) = 16.26 \). Forecast: \( \hat{Y}_5 = 25 + 16.26 \approx 41.26 \). The growing, always negative errors show that c = 10 is too large for a series that grows by 5 per period.

6. Differences: −10, 20, 20, 10 (t = 2 to 5). Forecast differences: 20 (t = 2, just the constant), \( 20 + 0.6 \times (-10) = 14 \), \( 20 + 0.6 \times 20 = 32 \), 32 and \( 20 + 0.6 \times 10 = 26 \). Adding them to the previous value, the forecasts are \( \hat{Y}_2 = 100 \), \( \hat{Y}_3 = 84 \), \( \hat{Y}_4 = 122 \), \( \hat{Y}_5 = 142 \) and \( \hat{Y}_6 = 146 \). Errors at times 2 to 5: −30, 6, −12 and −22; SSE = 1564. With \( n - 2 \) in the denominator, as in regression, \( SEE = \sqrt{1564/2} \approx 27.96 \).

7. \( \widehat{\nabla Y_2} = 20 \), \( \epsilon_2 = -30 \); \( \widehat{\nabla Y_3} = 20 + 0.6 \times (-10) + 0.3 \times (-30) = 5 \), \( \epsilon_3 = 15 \); \( \widehat{\nabla Y_4} = 20 + 12 + 4.5 = 36.5 \), \( \epsilon_4 = -16.5 \); \( \widehat{\nabla Y_5} = 20 + 12 - 4.95 = 27.05 \), \( \epsilon_5 = -17.05 \); \( \widehat{\nabla Y_6} = 20 + 6 - 5.115 = 20.885 \). Forecasts: \( \hat{Y}_2 = 100 \), \( \hat{Y}_3 = 75 \), \( \hat{Y}_4 = 126.5 \), \( \hat{Y}_5 = 137.05 \) and \( \hat{Y}_6 \approx 140.89 \).

이 기사가 어떠셨나요?

0^%

사용자 평점 (0)

이 기사를 공유하세요!