Material to accompany this classDownload the spreadsheet found here on Google Sheets. We will use this spreadsheet throughout the class. Download it in CSV format (Comma-Separated Values), the most common format for storing a data table that can be read by any language.Create a Python 3 development environment or use a ready-made online environment such as JupyterLab online, and load the file into it.The numbers in the spreadsheet use a decimal comma, as in 0,0177 (Brazilian notation). What happens if you read the CSV with pd.read_csv without the argument decimal=','?Answer: The comma columns are read as text, and .mean() then raises an error — pandas assumes a dot as the decimal separator; a value like 0,0177 becomes the string '0,0177', and in current pandas the mean of that column raises a TypeError. pandas does not guess the decimal separator: you have to tell it.Pandas in PythonIf you use the online version, your folder will look like this:Folder in JupyterLab onlineWe will use pandas to understand our dataset. To import it (together with numpy and matplotlib, which we will use later), use:import pandas as pdimport numpy as npimport matplotlib.pyplot as pltAnd we will open the dataset with read_csv:df = pd.read_csv("Trabalho de Estatística 1 - Set de dados.csv", decimal=',')Remember to use decimal="," or pandas will not recognize values written with a comma as numbers: by default, Python assumes the dot as the decimal separator, as written in the United States, not the comma used in Brazil and much of Europe.Now use .info() or .describe() on the table to get a quick summary of each column (count of non-null values, mean, standard deviation, minimum, quartiles and maximum):Example with .describe()In the following examples we will work with column "22" (column = "22"). To calculate the mean, the median and the mode, use .mean(), .median() and .mode() respectively:Calculating the mean, mode and median in pandasNote the error in the image: .mean() and .median() return a single number, but .mode() always returns a list of values (a Series), because there may be more than one mode. That is why the :.2f format inside the f-string, which only works for a single number, fails with a TypeError. To show all modes, use:moda_total = df[column].mode()print("Mode(s):", moda_total.tolist())This column has two modes (1,033,106,135 and 1,766,383,450). In continuous data like these, the mode is usually not very informative: they are just values that happened to repeat.The standard deviation and the coefficient of variation (CV = standard deviation ÷ mean) can be calculated like this:Standard deviation and CV in pandaspandas' .std() uses the sample standard deviation (it divides by \( n-1 \)). A CV of 301% shows that the data are extremely spread out relative to the mean.Now let's calculate the skewness:Skewness in PandasThe formula used by pandas' .skew() is the adjusted Fisher-Pearson coefficient of skewness for samples:\[ G_1 = \frac{n}{(n-1)(n-2)} \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{s} \right)^3 \]Where:\( n \) = number of observations (missing values, NaN, are ignored);\( x_i \) = each individual value;\( \bar{x} \) = sample mean;\( s \) = sample standard deviation (with \( n-1 \) in the denominator).The factor \( \frac{n}{(n-1)(n-2)} \) corrects the bias of the measure in small samples. The second cell in the image uses a different measure, Pearson's coefficient of skewness, \( 3(\bar{x} - \text{median})/s \), which compares the mean and the median. The two measures give different numbers (4.80 and 0.96 in this column), but they agree on the sign: the distribution is right-skewed. The 0.4 thresholds used in the image are just a rule of thumb for deciding whether the skewness matters.Now the kurtosis calculation:Kurtosis in pandaspandas' .kurtosis() returns the excess kurtosis (also with a small-sample correction), that is, it already subtracts the kurtosis of the normal distribution:\[ G_2 = \frac{n(n+1)}{(n-1)(n-2)(n-3)} \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{s} \right)^4 - \frac{3(n-1)^2}{(n-2)(n-3)} \]That is why the reference is 0, not 3: a normal distribution has an excess kurtosis of 0, positive values indicate a leptokurtic distribution (heavy tails) and negative values a platykurtic one. The value 26.40 shows tails much heavier than those of the normal.Now let's draw the distribution with a histogram:Use matplotlib to create the histogram in PythonThe "if 'kde' in locals()..." block in the image can be deleted: no kde variable was ever created, so it never runs. The final chart:Chart built by matplotlibWhat curve shape do you see? I see a reverse J curve: the highest frequency is at the start (near zero) and the frequency drops quickly, with a long right tail. Notice also how the mean (286.8 million) is far above the median (9.9 million), pulled up by the extreme values.You can also draw a boxplot to look at your data:Boxplot in matplotlibThe dropna in the image is needed because matplotlib's boxplot does not ignore missing values. Note, however, that a boxplot is of little help with reverse J curves: the box is squeezed near zero and dozens of points show up as outliers.In column 22, the mean is 286.8 million and the median is 9.9 million. What does this difference indicate?Answer: Right skew, with a long tail of high values pulling the mean up — the median depends only on the central position, while the mean is dragged by extreme values. With left skew it would be the opposite: the mean below the median.The pandas command df[column].kurtosis() returned 26.40. How should this number be interpreted?Answer: Leptokurtic: pandas measures excess kurtosis, whose normal reference is 0, and 26.40 indicates very heavy tails — the most common mistake is to compare the pandas result with 3, which is the reference for kurtosis without the subtraction.Transforming your dataSince the curve has a reverse J shape, we can transform the data to bring it closer to a bell-shaped curve, which is easier to work with. In this case I will use the logarithm (np.log, which requires the import numpy as np done at the beginning):Logarithm transformation in pandasThe transformation can be done in several ways: we can take the logarithm, the square root, among others. Here I chose the logarithm because the right tail is very pronounced and the column has no zeros or negative values (the logarithm of zero is minus infinity). We usually apply the logarithm to financial values (salaries, stock prices), population sizes, system response times and sales data with positive outliers. The square root, on the other hand, usually suits counts: number of website visits, accidents, products sold and other event frequencies, including when there are zeros.Histogram of the log-transformed distributionJudging by the histogram, it looks like we got the transformation right!Boxplot of the log-transformed dataThe numbers confirm it: after the logarithm, the mean (16.14) and the median (16.10) practically coincide, the standard deviation is 3.03, the skewness drops from 4.80 to −0.03 (almost symmetric) and the excess kurtosis drops from 26.40 to −0.49 (slightly platykurtic, that is, close to the normal). For comparison, the square root still leaves a skewness of 2.64 in this column: here the logarithm really was the better choice.Column 4 of the spreadsheet has 25 values equal to zero. What happens if you apply np.log to this column?Answer: The zeros become minus infinity, and the means and charts of the transformed column become invalid — the logarithm of zero does not exist. In such cases use the square root or np.log1p, which computes the logarithm of 1 + x.After the logarithm, column 22 has a skewness of −0.03 and an excess kurtosis of −0.49. What does this indicate?Answer: That the distribution became practically symmetric, with tails close to those of the normal — skewness near 0 means symmetry, and an excess kurtosis of −0.49 is only slightly platykurtic. Left skew would require a clearly negative value, such as −1.Assignment: do it yourself!Choose a column of the dataset and answer the following questions:What is the mean of the distribution?What is the median of the distribution?Does the distribution have a mode?What is the standard deviation of the distribution?What is the coefficient of variation of your dataset?Draw the frequency distribution chart.What is the shape of the curve? (bell-shaped, J-shaped, reverse J, U-shaped, uniform, bimodal, trimodal, multimodal…)Looking at the values of the variables, what kind of data could they represent?Draw the boxplot of your distribution.
ردود الفعل
0
0 تعليقات
كن أول من يعلق
لا شيء هنا بعد.