IntroductionThe direct translation of the term outlier is "outside the line." An outlier is any data point that is considered outside the line; in other words, contaminating, strange, extreme, or aberrant observations for that data set.> Observations that show a large distance from the others or are inconsistent with them are usually called outliers.What is an outlier in real life?In many cases, the reasons for their existence determine how they should be treated. Thus, the main causes that lead to the appearance of outliers are:- Measurement errors;- Execution errors;- Inherent variability of the elements of the populationThey allow analysis of:- Fraud detection. [3] (Benford's law [4])- Consumer spending behavior.- In medical analyses (unexpected treatment results).- Pharmaceutical research.- Marketing.What to do with outliers?Concern about outlier observations is old and dates back to the first attempts to analyze a data set. Initially, it was thought that the best way to deal with this type of observation would be to eliminate it from the analysis.Opinions were not unanimous: some defended the rejection of observations “inconsistent with the others,” while others stated that observations should never be rejected simply because they appear inconsistent with the rest of the data and that all observations should contribute with equal weight to the final result.How to identify an outlierOutlier identification can go through three phases:The initial phase is the identification of observations that are potentially aberrant. Outlier identification consists of detecting surprising observations using subjective methods. Identification is generally done through graphical analysis or, if the number of data points is small, by direct observation of them. Thus, observations are identified that have a strong possibility of being designated as outliers.In the second phase, the objective is to eliminate the subjectivity inherent in the previous phase. The aim is to know whether the observations identified as potential outliers actually are outliers. Tests are performed on the “concerning” observation or observations. The most appropriate tests for the situation under study should be chosen. The suspicious observations are tested for their discordance. If the hypothesis that some observations are outliers is accepted, they can be designated as discordant. An observation is said to be discordant if it can be considered inconsistent with the remaining values after the application of an objective statistical criterion. Often the term discordant is used as a synonym for outlier.In the last phase, it is necessary to decide what to do with the discordant observations. The simplest way to deal with these observations is to eliminate them. As already stated, this approach, although widely used, is not advisable. It is only justified if the outliers are due to errors whose correction is unfeasible. Otherwise, the observations considered outliers should be treated carefully because they contain relevant information about characteristics underlying the data and may be decisive in understanding the population to which the sample under study belongs.For the last phase, several methods can be used:- Box plot- Discordance models- Dixon test- Grubbs test- Z-scores- etc...For this course, we will use the z-scores method to identify outliers. To be considered an outlier, the value must be greater than three standard deviations plus the mean or less than the mean minus three standard deviations. (Three-sigma rule [1])\[ \text{outlier} < \bar{x} - 3 * \sigma \]and:\[ \text{outlier} > \bar{x} + 3 * \sigma \]ExampleThe following values refer to nitrite concentrations in a water sample from a river: 0.403, 0.410, 0.401, and 0.380. The last observation is suspicious: should it be considered an outlier? The mean for this data set is: 0.398The standard deviation for this data set is: 0.011The mean plus three standard deviations is equal to 0.432. In other words, any value above 0.432 is considered an outlier.The mean minus three standard deviations is equal to 0.364. In other words, any value below 0.364 is considered an outlier.Since there is no value either above or below three standard deviations, there are no outliers for this data set.Exercises1) The following data refer to rainfall (in mm) in a given city over 5 months: 53.5, 61.5, 62.3, 64.9, 40.6. The standard deviation has already been calculated and was 8.84. Can any of the values mentioned above be considered an outlier? In the sixth month, the rainfall value was 30 mm; is it an outlier?2) The following values refer to wheat production: 12.0, 12.4, 17.5, 11.8, 14.0, 12.8, 14.0, 13.5, 12.6, 13.0, 12.6, 12.7. Its standard deviation is 1.38. Can any of the values mentioned above be considered an outlier?3) Consider the following hemodialysis times (in months) in 14 transplant patients: 51, 24, 55, 75, 24, 27, 22, 23, 48, 18, 96, 24, 26, and 35. Its standard deviation is 22.38. Check whether any of these observations can be considered an outlier.Answer Key1) No. Yes.2) Yes, 17.5.3) No.References[1] https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule[2] http://www.estgv.ipv.pt/PaginasPessoais/psarabando/CET%20%20Ambiente%202008-2009/Slides/8.%20Outliers.pdf[3] https://www.sciencedirect.com/science/article/pii/S1467089515300324[4] https://en.wikipedia.org/wiki/Benford%27s_law
Sign in and join the conversation
— Comments 0
, Reactions 1
Be the first to comment