Statistics in Python: Using Pandas to understand your distribution

•0•0Comment iconComment iconComment iconComment icon

Discover how to analyze statistical data in Python using the powerful Pandas library. Learn to interpret and visualize the distribution of your data to gain better insights and facilitate decision-making.

Writer image

Translated byEditorial

Writer image

Revised byEditorial

Edit Article

Material to accompany this class

Download the spreadsheet found here on Google Sheetslink outside website. We will use this spreadsheet for the class. Download it as CSV, the most common method for storing a database that can be read by any language (Comma-Separated Values).

Create a python3 development environment or use ready-made online environments such as JupyterLab onlinelink outside website and import the database.

Pandas in Python

Your folder will look like this if you use the online version:

Folder in JupyterLab online
Folder in JupyterLab online

As you can see, we will use pandas to understand our database. To import pandas, use:

import pandas as pd

And we will open our database using read_csv:

df = pd.read_csv("Trabalho de Estatística 1 - Set de dados.csv", decimal=',')

Remember to use decimal="," or it will not understand numbers separated by commas as numbers, since Python generally assumes that we write like people in the United States and not in Brazil.

Now use .info() or .describe() on the database so it provides us with quick data about its distribution:

Example with .describe()
Example with .describe()

To calculate the mean, mode, and median, use .mean(), .mode(), and .median(), respectively:

Calculating the mean, mode, and median in pandas
Calculating the mean, mode, and median in pandas

Notice that it tells you if there is more than 1 mode.

Standard deviation and Coefficient of Variation (CV) can be calculated in the following ways:

Standard deviation and CV in pandas
Standard deviation and CV in pandas

Now let's move on to calculating Skewness:

Skewness in Pandas
Skewness in Pandas

The formula used by Pandas to calculate skewness is:

\[ G_1 = \frac{n}{(n-1)(n-2)} \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{s} \right)^3 \]

Where:

- \( n \) = number of observations

- \( x_i \) = each individual value

- \( \bar{x} \) = sample mean

- \( s \) = sample standard deviation (with \( n-1 \) degrees of freedom)

The `.skew()` function was implemented following Fisher's approach and is a different formula for calculating skewness compared to Pearson, with:

- Correction for bias in small samples

- Consistency with other popular implementations

- Proper handling of missing values (NaN)

Now the calculation of kurtosis:

Kurtosis in pandas
Kurtosis in pandas

Now let's make the curve using a histogram:

Use matplotlib to create the histogram in python
Use matplotlib to create the histogram in python

The final chart:

Chart built with matplotlib
Chart built with matplotlib

What curve can you see? Here I see a J-shaped curve.

You can also make a boxplot to see your data:

Boxplot in matplotlib
Boxplot in matplotlib

Although boxplot is not the most recommended with J-shaped curves.

Transforming your data

Since the curve looks J-shaped, we can make transformations so that the curve becomes bell-shaped, a curve that is easier to work with. In my case, I will transform it using the logarithm:

Logarithm transformation in pandas
Logarithm transformation in pandas

Note that the transformation can be done in several ways: we can take the logarithm, we can take the square root. In this case, I found the logarithm appropriate because it has a very pronounced right tail and no zeros. We generally transform financial values (salaries, stock prices), population sizes, system response times, and sales data with positive outliers using logarithms. And we generally transform using the square root when there are: number of website visits, accident counts, number of products sold, and event frequency data.

Histogram of the log-transformed distribution
Histogram of the log-transformed distribution

Looking at the histogram, it seems we got the transformation right!

Boxplot of the Log-Transformed Data
Boxplot of the Log-Transformed Data

If we analyze the mean, standard deviation, skewness, and kurtosis, we can be more certain that we did well to transform it!

Assignment: do it yourself!

Choose a number to work with within the provided database and answer the following questions:

1) What is the mean of the distribution?

2) What is the median of the distribution?

3) Does the distribution have a mode?

4) What is the standard deviation of the distribution?

5) What is the coefficient of variation of your dataset?

6) Create the frequency distribution graph

7) What is the shape of the curve? (bell-shaped curve, J-shaped curve, U-shaped curve, uniform curve, bimodal, trimodal, multimodal……)

8) Analyzing the values of the variables, what possible data could they represent?

9) Make the boxplot of your distribution

Send your news

Topics

pandas distribuição programação
Telegram logo

Hi, we have a community in Whatsapp! Come talk to us about Statistic here!