Material to accompany this lessonDownload the spreadsheet found here on Google Sheets, with the daily prices of 20 stocks and gold from August 2014 to August 2017. Download it as CSV (Comma-Separated Values), the most common format for storing a data table that any language can read.Create a Python 3 development environment or use a ready-made online environment such as JupyterLab online, and import the database.Statsmodels in PythonRename the downloaded file to "dados-2.csv" to make it easier to work with. We will use the pandas, numpy, statsmodels and matplotlib packages. Below you can see how to import each one:Importing packages in PythonTime as the indexNotice that we use decimal="," and thousands="." because the numbers in the spreadsheet are in Brazilian format. Now let's use time (the Tempo column) as the table's index and create a "Dias" (days) column: calendar days since the first trading day. It will be the explanatory variable of the linear regression: its slope tells us how much the price of each asset rises or falls, on average, per day.Transformation to use time as an indexTwo remarks. df.sort_index() on its own does nothing, because the result is not stored, and in pandas 3.x inplace=True raises a deprecation warning; prefer df = df.set_index('Tempo').sort_index(). And asfreq("D") creates one row for each calendar day: weekends and holidays become NaN, and the table goes from 756 trading days to 1,097 days.Correlation matrixBefore the regressions, a very important analysis for investors: the correlation matrix, which shows how much the price of each asset moves together with every other one. To build a portfolio, prefer combining assets with low or negative correlation with each other: when one falls, the other tends not to fall with it, which lowers the risk. Below is the code to calculate the matrix:Calculating correlationsThe matrix should look like this at the end:Correlation matrix between stocksBelow is what Python printed. Since the matrix is symmetric, each pair appears twice. "Days" is not a company: the correlation with Days measures the asset's upward (positive) or downward (negative) trend in the period. Most positive correlations:Company 1Company 2CorrelationBradescoTim0.957497TimBradesco0.957497Domino's pizzaDays0.948813DaysDomino's pizza0.948813MastercardDomino's pizza0.921368Domino's pizzaMastercard0.921368SantanderAmerican Express Company0.888684American Express CompanySantander0.888684TimSantander0.888301SantanderTim0.888301Most negative correlations:Company 1Company 2CorrelationDaysSantander-0.642465SantanderDays-0.642465Ford motorsBraskem-0.668737BraskemFord motors-0.668737Ford motorsMastercard-0.708027MastercardFord motors-0.708027Domino's pizzaFord motors-0.826345Ford motorsDomino's pizza-0.826345Ford motorsDays-0.892252DaysFord motors-0.892252Between trending prices, much of the correlation comes simply from both rising or falling over time. For portfolio building, the correlation of daily returns is more appropriate: df.pct_change().corr().You want to build a two-stock portfolio with the lowest possible risk. Which pair in the table is the best starting point?Answer: Domino's pizza and Ford motors — with a correlation of -0.83, when one falls the other tends to rise, and the swings offset each other. Bradesco and Tim (0.96) move almost together and barely diversify.Simple linear regressionNow let's actually use Statsmodels. Below is the code for a simple linear regression between gold and the number of days. sm.add_constant() adds the intercept, which Statsmodels' OLS does not include on its own.Linear regression in StatsmodelsLinear regression provides several results. Look at the table below and try to interpret each parameter:OLS resultsThe Days coefficient (0.0255) says gold rose on average 0.026 per day (about 9.3 per year), and the R² of 0.308 says time explains only 31% of the price variation. The most worrying number is the Durbin-Watson of 0.015: the ideal is close to 2, and close to zero indicates strongly autocorrelated residuals, which makes the standard errors and p-values too optimistic. In addition, the 1,097 observations include the interpolated weekends; the real data are 756.Below are our forecast and the observed real-world points:Forecast vs observedNot one of the best regressions, right? The price rises and falls in waves that a straight line cannot follow. Let's also look at the residuals. In the histogram below, the horizontal axis is the residual value and the vertical axis the count of days; the labels in the code are swapped.The residuals are not symmetrical: long right tailIn the regression of gold on days, the Durbin-Watson was 0.015. What does this indicate?Answer: That the residuals are strongly autocorrelated, and the p-values are too optimistic — Durbin-Watson ranges from 0 to 4 and is close to 2 without autocorrelation. Near 0, one day's error practically repeats the previous day's, which violates an OLS assumption.ARIMA modelIf our friend linear regression cannot forecast, we can look for other models. Let's try an ARIMA(1,1,1) for the gold price. Unlike the regression, ARIMA does not use time as an explanatory variable: it explains the price by the series' own past (1 autoregressive term, 1 difference and 1 moving-average term).Python code for ARIMAARIMA resultForecast vs Observed with ARIMAThe red line sticks to the points, but it is a one-step-ahead forecast: each value is predicted already knowing the real price of the previous day, and for price series that is always close to the real value. The drop to zero at the start is not a data error either: on the first day the model has no past and returns zero. That is why we must not forget to look at the residuals (the filter abs(x) less than 40 discards that first residual, of about 86):ARIMA residualsThe residuals are centred on zero and the Ljung-Box test (Prob(Q) = 0.76) finds no remaining autocorrelation, but the kurtosis of 9.08 shows tails heavier than those of the normal distribution. In the chart, the horizontal axis is the residual, not time as the label says.In the ARIMA chart, the predicted line almost coincides with the observed points. What is the correct interpretation?Answer: Each fitted value uses the real price of the previous day, so it is a forecast only one day ahead — since today's price is very similar to yesterday's, any model that knows the previous day looks great on the chart. This says nothing about forecasting months ahead.Training set vs test setTo see whether the model works outside the sample, we split the data in half: we estimate the model with the training set and check how it performs on the test set. filter(model_fit.params) applies the whole series to the parameters estimated with the training set only.Creating the test and training sets and running a new regressionTraining ARIMA resultsFinal forecast:Test, Training, and observed valuesIt looks excellent, but the red line is still the one-step forecast, which on each test day receives the real price of the previous day. Redoing the calculations, its average error (root mean squared error) on the test set is 1.71, practically the same as saying tomorrow's price will be today's (1.72). The true forecast for the whole test period, without looking at its data, is model_fit.forecast(steps=len(teste)): it is almost flat, near 71, while gold went above 120, and the average error rises to 22.6. In other words, ARIMA(1,1,1) describes the series well day by day, but does not anticipate the big rises and falls.On the test set, ARIMA's one-step forecast had an average error of 1.71, and the rule 'tomorrow equals today' had 1.72. What do you conclude?Answer: ARIMA adds almost nothing to the naive rule of repeating yesterday's price — comparing with a naive model is the simplest way to know whether a time-series model has any value, and here the difference is 0.01.Multivariate regressionIf you do not want to work with ARIMA, you can run a multivariate regression, using stocks correlated with yours as explanatory variables. In the example below, I use Chevron's price and the number of days to explain the price of Vale (formerly Vale do Rio Doce, a Brazilian mining company).Python code for multivariate regressionMultivariate Regression resultR² rises to 0.72, but the Days coefficient has a p-value of 0.538: once Chevron is included, time adds nothing. The Durbin-Watson of 0.025 repeats the warning: between trending prices, a high R² may be a spurious regression.Forecast and observed values from the Multivariate RegressionIn the chart, the title and the vertical axis kept the labels of the gold example; what is shown there is Vale's price.In the regression of Vale on Days and Chevron, the Days coefficient had a p-value of 0.538. What should you do with this variable?Answer: It can be removed, because it adds no explanatory power after Chevron — with a p-value of 0.538 you cannot reject that the coefficient is zero. This does not mean Vale did not change over time: its variation is already captured by Chevron.ExercisesNow you do it:1) Import this dataset into pandas.2) Create the correlation matrix.3) Choose a stock and run a linear regression between it and the days. It cannot be gold!4) Choose a stock and fit an ARIMA to it.5) Choose a stock, train an ARIMA with half of the data and test it with the other half. Compare the error with that of the rule 'tomorrow equals today'.6) Challenge: run a multivariate regression that tries to explain the value of a stock using time and the value of another stock as a reference.
ردود الفعل
0
0 تعليقات
كن أول من يعلق
لا شيء هنا بعد.