IntroductionThe word outlier comes from out + lie: something that "lies outside" the rest. In everyday language we might call it an atypical value, a discrepant value or something "off the charts". An outlier is an observation that is contaminating, strange, extreme or aberrant compared with the rest of the data set.“Observations that lie far away from the others, or that are inconsistent with them, are usually called outliers.”- Definition of outlier [2]What is an outlier in real life?Often, the reason an outlier exists determines how it should be handled. The main causes of outliers are:measurement errors;execution errors (recording, typing, procedure);the inherent variability of the elements of the population — the value is rare, but true.Looking for outliers is also useful in its own right. They allow us to analyze, for example:fraud detection [3] (see also Benford's law [4]);consumer spending behavior;medical analyses (unexpected treatment results);pharmaceutical research;marketing.What should we do with outliers?Concern about outliers is old and dates back to the first attempts to analyze a data set. At first, people thought the best way to deal with this kind of observation was to remove it from the analysis.Opinions were never unanimous: some advocated rejecting observations "inconsistent with the rest", while others argued that no observation should be rejected just because it seems inconsistent with the rest of the data, and that all of them should contribute with the same weight to the final result.A thermometer recorded 412 °C for a patient's temperature in a hospital. What should be done with this value?Answer: Correct it if the right value can be known (probably 41.2) and, if not, remove it — it is a typing error (412 °C is physically impossible), and errors that cannot be corrected are the only case in which removal is justified. Always keeping everything is the tempting answer, but it applies to rare true values, not to errors.How to identify an outlierIdentifying outliers can involve three phases.The first phase is identifying the observations that are potentially aberrant. It is done with subjective methods: usually by graphical analysis or, if the data set is small, by looking directly at the values. This points out the observations that have a strong chance of being considered outliers.In the second phase, the goal is to remove the subjectivity of the previous phase: we want to know whether the observations flagged as potential outliers really are outliers. To do so, we apply an objective statistical criterion or test to the suspicious observations, chosen according to the situation under study. If the criterion indicates that the observation is inconsistent with the other values, it is called discordant. The term discordant is often used as a synonym for outlier. Some criteria used in this phase:the box plot and the interquartile range rule;z-scores (the three-standard-deviation rule);Dixon's test;Grubbs' test;discordancy models, among others.In the last phase, we must decide what to do with the discordant observations. The simplest way is to remove them but, as already mentioned, this approach, although widely used, is not advisable. It is only justified when the outliers come from errors that cannot be corrected. Otherwise, observations considered outliers should be handled with care, because they carry relevant information about the data and may be decisive for understanding the population the sample came from.In this course we will use two criteria: the z-score and the interquartile range rule.Criterion 1: z-score (three-standard-deviation rule)The z-score measures how many standard deviations a value is away from the mean:\[ z = \frac{x - \bar{x}}{s} \]By the three-standard-deviation rule (also called the three-sigma rule [1]), a value is considered an outlier when it is more than three standard deviations away from the mean, that is, when \(|z| > 3\):\[ x < \bar{x} - 3s \quad \text{or} \quad x > \bar{x} + 3s \]Here \(s\) is the sample standard deviation (dividing by \(n-1\)), as we saw in the lesson on sampling. The justification comes from the normal distribution: in it, about 99.7% of the values lie less than three standard deviations from the mean.Beware of small samples. With \(n\) observations, no value can have \(|z|\) greater than \((n-1)/\sqrt{n}\), because the outlier itself pulls the mean and inflates the standard deviation. With 4 values, the maximum is 1.5; with 10 values, 2.85. Only from 11 observations on can the three-standard-deviation rule flag any outlier at all. In small samples, use another criterion (such as the interquartile range rule or Dixon's test).Example: the following values are nitrite concentrations in a river water sample: 0.403, 0.410, 0.401 and 0.380. The last observation is suspicious: should it be considered an outlier?The mean of this data set is 0.3985, and the sample standard deviation is 0.0129.The mean plus three standard deviations is 0.3985 + 3 × 0.0129 ≈ 0.437. Any value above 0.437 would be considered an outlier.The mean minus three standard deviations is 0.3985 − 3 × 0.0129 ≈ 0.360. Any value below 0.360 would be considered an outlier.Since no value is above 0.437 or below 0.360, there are no outliers by this criterion: the z-score of 0.380 is only −1.43. Note that, with only 4 values, this result was unavoidable (the maximum possible is 1.5).A data set has mean 50 and sample standard deviation 4. By the three-standard-deviation rule, which of these values would be an outlier?Answer: 63 — the limits are 50 − 3 × 4 = 38 and 50 + 3 × 4 = 62, and only 63 falls outside them (z = 3.25). 38.5 looks extreme, but it is still above 38 (z = −2.875).Criterion 2: interquartile range rule (box plot)This is the criterion used in the box plot, proposed by John Tukey in 1977. We compute the first quartile \(Q_1\), the third quartile \(Q_3\) and the interquartile range \(IQR = Q_3 - Q_1\). A value is an outlier if it lies outside the "fences":\[ x < Q_1 - 1.5 \times IQR \quad \text{or} \quad x > Q_3 + 1.5 \times IQR \]Since the quartiles do not depend on the extreme values, an outlier cannot "hide itself" the way it does with the z-score. There are small variations in how quartiles are computed; here we use the one in Excel (QUARTILE or QUARTILE.INC) and numpy, with linear interpolation.Example: the delivery times (in days) of 9 orders were 3, 4, 4, 5, 5, 6, 6, 7 and 15. Is the 15-day order an outlier?The quartiles are \(Q_1 = 4\) and \(Q_3 = 6\), so \(IQR = 2\). The fences are at 4 − 1.5 × 2 = 1 and 6 + 1.5 × 2 = 9. Since 15 > 9, the 15-day order is an outlier.By the z-score it would not be: the mean is 6.11 and the sample standard deviation is 3.55, so \(z = (15 - 6.11)/3.55 \approx 2.50\), below 3 — and, with 9 values, no z could exceed 2.67.In a data set, Q1 = 20 and Q3 = 30. By the interquartile range rule, what is the upper fence, above which a value is an outlier?Answer: 45 — the IQR is 30 − 20 = 10, and the upper fence is Q3 + 1.5 × IQR = 30 + 15 = 45. 35 is the most common wrong calculation: adding 1.5 × IQR to the median or using only half the IQR.Exercises1) The following data are the rainfall (in mm) in a certain city over 5 months: 53.5, 61.5, 62.3, 64.9, 40.6. The sample standard deviation is 9.89. By the three-standard-deviation rule, can any of these values be considered an outlier? In the sixth month, rainfall was 30 mm: compared with the five previous months, is this value an outlier?2) The following values refer to wheat production: 12.0, 12.4, 17.5, 11.8, 14.0, 12.8, 14.0, 13.5, 12.6, 13.0, 12.6, 12.7. The sample standard deviation is 1.51. Is any of these values an outlier (a) by the three-standard-deviation rule? (b) By the interquartile range rule?3) Consider the following hemodialysis times (in months) of 14 transplant patients: 51, 24, 55, 75, 24, 27, 22, 23, 48, 18, 96, 24, 26 and 35. The sample standard deviation is 23.23. Check whether any of these observations is an outlier (a) by the three-standard-deviation rule; (b) by the interquartile range rule.Answer key1) No and no. The mean of the five months is 56.56, and the limits are 56.56 ± 3 × 9.89, that is, from 26.9 to 86.2. None of the five values falls outside this interval (and, with n = 5, none could), and 30 mm is also inside it. Note: if you use the population standard deviation (8.84), dividing by n, the lower limit would be 30.03 and 30 mm would fall outside by a hair — a sign that, with so little data, the conclusion is fragile.2) (a) No. The mean is 13.24 and the upper limit is 13.24 + 3 × 1.51 = 17.77; the z-score of 17.5 is 2.82. (b) Yes, 17.5. Q1 = 12.55, Q3 = 13.625, IQR = 1.075, and the upper fence is 13.625 + 1.5 × 1.075 ≈ 15.24.3) (a) No. The mean is 39.14 and the upper limit is 39.14 + 3 × 23.23 = 108.8. (b) Yes, 96. Q1 = 24, Q3 = 50.25, IQR = 26.25, and the upper fence is 50.25 + 1.5 × 26.25 ≈ 89.6.References[1] https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule[2] http://www.estgv.ipv.pt/PaginasPessoais/psarabando/CET%20%20Ambiente%202008-2009/Slides/8.%20Outliers.pdf[3] https://www.sciencedirect.com/science/article/pii/S1467089515300324[4] https://en.wikipedia.org/wiki/Benford%27s_law
ردود الفعل
0
0 تعليقات
كن أول من يعلق
لا شيء هنا بعد.