Material to accompany this lessonDownload the spreadsheet found here on Google Sheets, which we will use in the lesson. Download it as CSV (Comma-Separated Values), the most common format for storing a data table that any language can read.Create a Python 3 development environment or use a ready-made online environment such as JupyterLab online, and import the database.Scipy in PythonRename the downloaded file to "dados-3.csv" to make it easier to work with. We will use the pandas, numpy, statsmodels, scipy and matplotlib packages. The scipy.stats module is the one that provides probability distributions (normal, t, binomial...) and statistical tests. Below you can see how to import each one:Importing packages in PythonTime as the indexNotice that we use decimal="," and thousands="." because the numbers in the spreadsheet are in Brazilian format. Now let's use time (the Tempo column) as the table's index and calculate the daily change in the price of gold (the Ouro column). I am using gold, but I recommend you also test other stocks. To measure the change we use .diff() (absolute change, in currency units) or .pct_change() (percentage change).Converting the Tempo column into the indexWatch out, because this code does not work as expected in current versions of pandas (3.x). The line df['Variacao_Percentual'].fillna(0, inplace=True) no longer changes the table (the warning in the image already anticipated this), and the first day remains empty (NaN). With that NaN, the normality test in the next section returns nan instead of the p-value. Besides, filling the first day with zero invents a change that never happened. It is more correct to drop the first day:df['Variacao_Percentual'] = df['Ouro'].pct_change() * 100variacao = df['Variacao_Percentual'].dropna()The inplace=True parameters of set_index and sort_index also raise deprecation warnings; prefer df = df.set_index('Tempo').sort_index(). The results change very little (the normality test p-value goes from 4.07e-07 to 4.43e-07), and the conclusions below still hold.Histogram of the gold price changeLet's see what our data look like. The best way to start is a histogram:Creating the gold price histogramThe result was:Histogram of the changeAt first glance, the histogram looks like a normal distribution: it is bell-shaped and almost symmetric around zero. But "looking like" is not enough; let's test it.Normality testTo check whether the data could come from a normal distribution, we use scipy's normaltest function (the D'Agostino and Pearson test, which combines skewness and kurtosis). The null hypothesis of this test is that the data come from a normal distribution.Normality test with ScipyThe p-value was 4.0693e-07, that is, about 0.00004%. If the changes were normal, the chance of getting data this far from normal would be tiny. So we reject the null hypothesis: gold's daily changes do not follow a normal distribution. The reason shows up in the numbers printed above: the skewness is practically zero (0.005), but the excess kurtosis is 1.60, meaning there are more days close to zero and more extreme days than the normal distribution predicts (heavy tails). The chart below shows this: the bars exceed the curve in the centre and at the ends.Actual distribution vs Normal distributionScipy's normaltest returned a p-value of 4e-07 for the daily changes. What is the correct conclusion at the 5% significance level?Answer: Normality is rejected: the data do not follow a normal distribution — the null hypothesis of normaltest is normality. A tiny p-value is evidence AGAINST the null hypothesis, that is, against normality.Identifying outliersEven without perfect normality, a distance of 3 standard deviations from the mean is a widely used rule of thumb for flagging anomalous days, called outliers. If the changes were normal, only 0.27% of days would fall outside this range. You can change the 3 to detect larger or smaller anomalies.Code to detect OutliersThe days detected as outliers for gold were:DayChange %2014-11-069.2123062014-11-28-8.1771722014-12-177.5590812015-10-027.8665082016-02-088.7100892016-05-04-10.8404772016-06-038.4374262016-10-04-8.7556682016-11-10-10.078868With 756 days, a normal distribution would produce about 2 outliers (0.27% of 756); we found 9. It is the same conclusion as the normality test seen from another angle: extreme days are much more common than the normal distribution predicts. If you like, look up in the news what happened on each of these days.If the changes were normal, about 0.27% of the 756 days would lie more than 3 standard deviations from the mean. 9 were found. What does this indicate?Answer: That the distribution has heavier tails than the normal — 0.27% of 756 gives about 2 expected days. Seeing 9 extreme days, more than four times the expected number, matches the excess kurtosis of 1.6 and the rejection of normality.Should we invest in gold?Do we have enough data to say that gold is rising? To find out, we build a confidence interval for the mean daily change. Even without normality, this is valid here: with 756 days, by the central limit theorem the mean of the changes is approximately normally distributed.Building the Confidence Interval with ScipyThe sample mean is positive (0.040% per day), but the 90% interval goes from −0.105% to 0.185% and includes zero. In other words, we do not yet have enough data to state that gold's mean change is greater than zero. A t-test of the mean against zero (stats.ttest_1samp(variacao, 0)) leads to the same conclusion, with a p-value of 0.65.The 90% interval for gold's mean daily change was −0.105% to 0.185%. What can be concluded?Answer: That there is not enough evidence that the mean change is different from zero — the interval includes zero, so zero is a plausible value for the mean. This does not prove gold does not rise; it only shows that 3 years of data are not enough to say so.What if we want more certainty: how many days do we need?Are 3 years of data enough? How many trading days would we need to estimate the mean daily change with a margin of error of only 0.1 percentage point, at 90% confidence? We use the sample size formula, n = (z · σ/E)²:Sample Size with ScipyThe result is n = (1.6449 × 2.4183/0.1)² ≈ 1,583 trading days, more than twice the 756 available. Since a year has about 252 trading days, this is a little over 6 years of data. And notice: even with this margin, if the mean stayed at 0.04%, the interval (0.04 ± 0.1) would still include zero. To separate a mean of 0.04% from zero, the margin would have to be smaller than 0.04, which requires about 9,900 trading days, almost 40 years.Estimating the mean change with a margin of 0.1 percentage point requires 1,583 trading days. How many would be needed for a margin of 0.05?Answer: About 6,330 — n grows with the square of 1/E. Halving the margin multiplies n by 4: 1,583 × 4 = 6,332.What should the exam theme be?
ردود الفعل
0
0 تعليقات
كن أول من يعلق
لا شيء هنا بعد.