Regression Models over Time: ARIMA and Time Series

•0•0Comment iconComment iconComment iconComment icon

In this class, we will explore how to model and forecast time series using ARIMA, including fundamental concepts, parameter selection, and performance evaluation. We will also see practical exercises with temporal data to apply the concepts learned.

Writer image

Translated byEditorial

Writer image

Revised byEditorial

Edit Article

The ARIMA (AutoRegressive Integrated Moving Average) is a statistical model widely used for time series analysis and forecasting. It combines three main components:

- AR (Autoregressive): Dependence on past values.

- I (Integrated): Differencing to make the series stationary.

- MA (Moving Average): Dependence on past errors.

Thus, ARIMA ends up creating a framework for handling time series much better than a simple linear regression. Let’s explain what ARIMA is and then compare it with a simple linear regression to understand why it is better when dealing with time.

Components of the ARIMA Model

1. AR(p) – Autoregressive of Order p

Autoregressive series are those that depend on their own past values; generally, we are talking about population size or product prices, inflation in general. In a population, the number of individuals grows over time, and the more individuals there are, the more it can grow. A product’s price depends on last year’s value plus inflation during the year, and note that inflation is a percentage of the product’s old value.

It models \( Y_t \) as a linear combination of its past values:

\[ Y_t = c + \phi_1 Y_{t-1} + \phi_2 Y_{t-2} + \dots + \phi_p Y_{t-p} + \epsilon_t \]

Where p is the number of lags used.

2. I(d) – Integration of Order d

Number of differences needed to make the series stationary, that is, without an upward or downward trend.

A time series is considered stationary when its statistical properties (such as mean, variance, and autocorrelation) do not change over time. This is crucial for models like ARIMA, which assume stationarity to make reliable forecasts. Use the Augmented Dickey-Fuller Test to check stationarity. This test will be covered later; here, you will do it visually.

Look at the following series and notice that working with the difference between values over time keeps the series with a constant mean:

Month (t)Sales (Yₜ)1st Difference (∇Yₜ)
1100–
2120120 - 100 = 20
3130130 - 120 = 10
4150150 - 130 = 20
5170170 - 150 = 20
6190190 - 170 = 20

Thus, we apply I(d) when we notice an upward or downward trend in your series. If the growth is not linear, we can use I(2) instead of I(1), that is, apply the difference again until we have a constant mean.

3. MA(q) – Moving Average of Order q

We use MA(q) when your series has sudden spikes and you need to correct for the errors only from that moment. We use MA(1) or MA(2) when the series has “short memory.” The value is affected by the previous error. Examples of data that suffer temporary impacts: stock or product prices after a one-time event, demand after a flash sale.

It models \( Y_t \) as a linear combination of past errors:

\[ Y_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + \dots + \theta_q \epsilon_{t-q} \]

- q: Number of error terms used.

- ACF: Identifies the order \( q \) (cutoff at the significant lag).

Day (t)Price (Yₜ)Error (εₜ)MA(2) Model Calculation
150.00+2.00-
252.00-1.00-
351.50Y₃ = 50 + 1.50 + 0.8×(-1) + 0.3×2 = 51.50
452.80-0.80Y₄ = 50 + (-0.80) + 0.8×1.5 + 0.3×(-1) = 52.80
553.34+0.34Y₅ = 50 + 0.34 + 0.8×(-0.8) + 0.3×1.5 = 53.34
652.72-0.28Y₆ = 50 + (-0.28) + 0.8×0.34 + 0.3×(-0.8) = 52.72

ARIMA(p, d, q) Notation

- ARIMA(1, 1, 1):

- AR(1): \( Y_t \) depends on \( Y_{t-1} \).

- I(1): 1st differencing applied.

- MA(1): \( Y_t \) depends on \( \epsilon_{t-1} \).

Example ARIMA(1,1,1) Equation:

\[ \nabla Y_t = c + \phi_1 \nabla Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \]

Practical Example: Monthly Sales Forecasting

Step 1: Load and Visualize the Data

Monthly sales data (in thousands):

MonthSales
1100
2120
3130
4150
5170
6190

Step 2: Check Stationarity

- ADF Test: p-value = 0.12 (non-stationary).

- Apply 1st differencing:

\[ \nabla Y_t = Y_t - Y_{t-1} \]

We can observe this in the table below:

MonthSales\( \nabla Y_t \)
212020
313020
415010
517020
619020

- New ADF test: p-value = 0.01 (stationary).

Step 3: Define the ARIMA(1,1,1) Model

The model formula is:

\[ \nabla Y_t = \phi_1 \nabla Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \]

Where:

- \(\nabla Y_t\) = Yₜ - Yₜ₋₁ (differenced series)

- \(\phi_1\) = AR(1) coefficient

- \(\theta_1\) = MA(1) coefficient

- \(\epsilon_t\) = error at time \(t\)

Step 4: Estimate the Coefficients (AR and MA)

Since we are doing this manually, we will use typical values for illustration:

- \(\phi_1 = 0.6\) (AR)

- \(\theta_1 = -0.3\) (MA)

Packages such as Python’s statsmodels use Markov Chain to arrive at the optimal constants by minimizing the function’s SSE.

Step 5: Calculate the Forecasts

Let’s forecast \(\nabla Y_t\) starting from month 3:

Month∇Yₜ (Actual)ARIMA(1,1,1) Forecast
310\(\hat{\nabla Y_3} = 0.6 \times 20 + (-0.3) \times \epsilon_2\)
420\(\hat{\nabla Y_4} = 0.6 \times 10 + (-0.3) \times \epsilon_3\)
520\(\hat{\nabla Y_5} = 0.6 \times 20 + (-0.3) \times \epsilon_4\)
620\(\hat{\nabla Y_6} = 0.6 \times 20 + (-0.3) \times \epsilon_5\)

Step 6: Calculate the Errors (\(\epsilon_t\))

Assuming \(\epsilon_1 = 0\) (initial error):

Month∇Yₜ (Actual)Forecast (\(\hat{\nabla Y_t}\))Error (\(\epsilon_t = \nabla Y_t - \hat{\nabla Y_t}\))
220-\(\epsilon_2 = 20 - 0 = 20\) (initial value)
310\(0.6 \times 20 + (-0.3) \times 20 = 6\)\(\epsilon_3 = 10 - 6 = 4\)
420\(0.6 \times 10 + (-0.3) \times 4 = 4.8\)\(\epsilon_4 = 20 - 4.8 = 15.2\)
520\(0.6 \times 20 + (-0.3) \times 15.2 = 7.44\)\(\epsilon_5 = 20 - 7.44 = 12.56\)
620\(0.6 \times 20 + (-0.3) \times 12.56 = 8.23\)\(\epsilon_6 = 20 - 8.23 = 11.77\)

Step 7: Forecast the Next Month (Month 7)

Using the latest values:

\[\hat{\nabla Y_7} = 0.6 \times 20 + (-0.3) \times 11.77 = 8.47\]

\[ \hat{Y_7} = Y_6 + \hat{\nabla Y_7} = 190 + 8.47 = 198.47\]

Summary of the ARIMA(1,1,1) Model

- Estimated equation:

\[ \nabla Y_t = 0.6 \nabla Y_{t-1} - 0.3 \epsilon_{t-1} + \epsilon_t \]

- Forecast for month 7: ~198.5 (rounded)

Difference Between ARIMAs

ModelFormulaWhen to Use?Practical Example
ARIMA(0,0,0)
(White Noise)		\( Y_t = \mu + \epsilon_t \)When the series is pure noise with no patternsControlled room temperature
ARIMA(0,1,0)
(Random Walk)		\( Y_t = Y_{t-1} + \epsilon_t \)Series with a stochastic trendStock prices
ARIMA(1,0,0)
(AR(1))		\( Y_t = c + \phi_1 Y_{t-1} + \epsilon_t \)When Yₜ depends only on Yₜ₋₁Daily energy demand
ARIMA(0,0,1)
(MA(1))		\( Y_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} \)Temporary shocks affect only 1 periodPost-promotion sales
ARIMA(1,0,1)
(ARMA(1,1))		\( Y_t = c + \phi_1 Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \)When there is memory of the past value AND the past errorCommodity prices
ARIMA(1,1,0)\( \nabla Y_t = \phi_1 \nabla Y_{t-1} + \epsilon_t \)Trend + autocorrelation at the 1st lagGDP growth
ARIMA(0,1,1)\( \nabla Y_t = \theta_1 \epsilon_{t-1} + \epsilon_t \)Trend + temporary shocksOil prices
ARIMA(1,1,1)\( \nabla Y_t = \phi_1 \nabla Y_{t-1} + \theta_1 \epsilon_{t-1} + \epsilon_t \)Combines all effectsSales with seasonality

Why Use ARIMA Instead of Linear Regression with Time?

The choice between ARIMA and linear regression with time depends on the structure of the data and the statistical assumptions. Linear Regression with Time (Simple Model) assumes that the time series can be modeled as:

\[ Y_t = \beta_0 + \beta_1 \cdot t + \epsilon_t \]

Where:

- \( t \) = time variable (e.g., 1, 2, 3, ...).

- \( \epsilon_t \) = random error.

The advantage is that it is simple to implement and interpret. It is also useful when there is a clear linear trend (e.g., constant growth). But it ignores autocorrelation: In time series, past values influence future values (e.g., product demand, stock prices); linear regression treats each observation as independent, which is rarely true in temporal data. It also does not handle seasonality or cycles; if the data has repetitive patterns (e.g., Christmas sales), linear regression does not capture them. And it requires uncorrelated residuals: in time series, residuals often have autocorrelation (e.g., if today was a high-demand day, tomorrow may be too).

Direct Comparison: Linear Regression vs. ARIMA

CriterionLinear Regression with TimeARIMA
TrendModels only linearRemoves trend through differencing
AutocorrelationIgnoresCaptures (AR and MA)
SeasonalityDoes not handleModels (SARIMA)
ResidualsRequires independenceAllows controlled autocorrelation
ComplexityLowHigh (choice of p, d, q)

Exercises

1. What is the difference between the AR and MA components in an ARIMA?

a) AR uses past values, MA uses past errors.

b) AR is for trend, MA is for seasonality.

c) Both are the same.

2. Make a forecast for time 6 using ARIMA(0,1,0).

MonthSales
1100
2120
3130
4150
5170
6190

3. Fit an ARIMA(1,0,0) to the series below and make a forecast for t=5 with \(\phi_1 = 0.6\) and \(\theta_1 = -0.3\) and \(c=10\).

tY
110
215
320
425

4. Fit an ARIMA(1,0,1) to the series below and make a forecast for t=5 with \(\phi_1 = 0.6\) and \(\theta_1 = -0.3\) and \(c=10\).

tY
110
215
320
425

5. Fit an ARIMA(1,1,1) to the series below and make a forecast for t=5 with \(\phi_1 = 0.6\) and \(\theta_1 = -0.3\) and \(c=10\).

tY
110
215
320
425

6. Use ARIMA(1,1,0) to calculate the forecast for all times from 1 to 6 (do 1 extra time step). Use \(\phi_1 = 0.6\), \(\theta_1 = 0.3\) and \(c=20\). After that, calculate the SEE of the forecast using time 2 to 5 as a reference.

TimeY (Sales)
180
270
390
4110
5120

7. Use ARIMA(1,1,1) to calculate the forecast for all times from 1 to 6 (do 1 extra time step). Use \(\phi_1 = 0.6\), \(\theta_1 = 0.3\) and \(c=20\).

TimeY (Sales)
180
270
390
4110
5120

Answer Key

1. a)

Send your news

Topics

arima tempo regressao
Telegram logo

Hi, we have a community in Whatsapp! Come talk to us about Statistic here!