Calculating probabilities with Scipy in Python

14/07/26 •0•0

Discover how to use Scipy in Python to calculate probabilities efficiently and accurately. This comprehensive guide will walk you through the process step by step with real examples.

撰写者Leon

翻译者Editorial

修订者Editorial

Material to Follow This Lesson

Download the spreadsheet found here on Google Sheets. We will use this spreadsheet for the lesson. Download it as CSV, the most common method of storing a database that can be read by any language (Comma-Separated Values).

Create a python3 development environment or use ready-made online environments such as jupyter lab online and import the database.

Scipy in Python

Start by downloading the data as CSV, renaming it to "dados-3.csv" to make it easier to work with. We will use the pandas, numpy, statsmodels, scipy, and matplotlib packages in python. Below you can see how to import each one:

Importing packages in python

Time as an Index

Note that we are using decimal="," and thousands="." precisely because our data is in Portuguese. Now we will use time as the index of our data table. Below is the code to set time as the index.

Transforming the Time column into an index

After transforming the Time column into an index, we will check how much the price of Gold has increased over time. Note that I am using gold here, but I recommend you test other stock prices. To find out how it has grown over time, we use the .diff() or pct_change() functions.

Histogram of the Variation in the Price of Gold

Let's check what our data looks like. For this, the best way is to create a histogram chart as shown below.

Creating the histogram of the gold price

The result was:

Histogram of the variation

Note that this histogram resembles the normal distribution!

Normality Test

Since our data looks very much like a normal distribution, we can perform a normality test to verify the hypothesis that it is in fact a normal distribution. For this, we use scipy with the normaltest function.

Normality test with Scipy

The result gave a very small probability! A probability of 4.0693e-07. Could it be a coincidence? Of course it could, but the chance is so small that we can reject the null hypothesis that there is no normality. See below the graph of the normal distribution plotted with our data:

Real distribution vs Normal distribution

Identifying Outliers

From now on, let's consider that the daily variation in Gold prices follows a Normal distribution. With this, we can identify days with anomalies. These days with anomalies are called "outliers", very unlikely days. For this, we will use the distance of 3 standard deviations to characterize that a data point is very far from the mean and is in fact an anomaly. This 3-standard-deviation characteristic is widely used in the industry, but you can modify the value if you want to detect even larger anomalies.

Code to detect Outliers

The days detected as outliers for Gold were:

Day	Variation %
2014-11-06	9.212306
2014-11-28	-8.177172
2014-12-17	7.559081
2015-10-02	7.866508
2016-02-08	8.710089
2016-05-04	-10.840477
2016-06-03	8.437426
2016-10-04	-8.755668
2016-11-10	-10.078868

If you want, you can search the newspaper for each of these days. Did something different happen on those days?

Should We Invest in Gold?

Now that we know the distribution is Normal, do we have enough data to say that Gold is growing over time? For this, we need to build a confidence interval for the increase in Gold using our sample.

Building the confidence interval with Scipy

Note that although Gold has a positive sample mean, we still do not have enough data to confirm that the variation in Gold is above zero.

And If We Want to Be More Certain, How Many Samples Do We Need?

Imagine then that we want to verify whether Gold is in fact a safe investment. Is just 3 years of data enough? How many days do we need for our sample? Let's see below how we would do it to have a margin of error of only 0.1% in the increase, which can help us with the inference about this investment.