Hypothesis test calculator with z and t
Your data
Pick the direction before you look at the data, not after. Choosing to test only for a higher value because the average you measured happened to come out higher is the oldest way there is to turn a coin flip into a finding.
Results
How often a difference at least this big would turn up by chance alone, if the real average were exactly the value you are comparing against
—
| The test statistic | — |
| Which distribution that is read against | — |
| Degrees of freedom | — |
| The value the statistic has to beat | — |
| How big the difference is, counted in standard deviations | — |
| What the other distribution would have answered | — |
| Range the true average is compatible with | — |
The same difference, measured on more people
| Observations | The test statistic | How often chance alone would do it | Verdict |
|---|
Nothing about the difference changes down that table. The average, the value being compared against and the deviation all stay exactly where you put them, and only the number of observations grows. Yet the verdict flips partway down, because the test is measuring how well you measured, not how big the thing is. That is why the size of the difference sits in the table above, next to the answer.
The other line worth reading is the one comparing the two distributions. When the deviation was estimated from the sample rather than known in advance, the Student t is the honest one, and it is more demanding than the normal exactly because it has to allow for the deviation itself being a guess. On small samples the gap is wide enough to change the verdict, and it only closes slowly: even with a hundred observations the two still part company in the third decimal.
One thing this figure is not, however often it is described that way: it is not the chance that the value you compared against is the true one. It is calculated by assuming that value is true and asking how surprising your data would be. Turning that around needs something the data alone cannot supply, which is how plausible the value was before you measured anything.
Should I use the z test or the t test?
Use the t test whenever the standard deviation came from the same sample you are testing, which is nearly always. Use z only when the deviation is genuinely known in advance for the whole population, from a long history of the same measurement rather than from these observations.
It matters most on small samples. With the same statistic of 2.10 and a two sided test, the t gives 0.1037 on five observations while the normal gives 0.0357: one of them calls it a finding at 5% and the other does not. The two agree from about twenty observations onward, and even at a hundred they still differ in the third decimal.
| Observations | p by the t | p by the normal |
|---|---|---|
| 5 | 0.1037 | 0.0357 |
| 8 | 0.0739 | 0.0357 |
| 10 | 0.0651 | 0.0357 |
| 15 | 0.0543 | 0.0357 |
| 20 | 0.0493 | 0.0357 |
| 30 | 0.0445 | 0.0357 |
What does the p value actually mean?
It is how often a difference at least as big as yours would show up by chance alone, in a world where the reference value is the true average. Nothing more.
It is not the probability that the reference value is true, and it is not the probability that your result was a fluke. Both readings are common and both are wrong, because the calculation starts by assuming the reference value is true, so it can never tell you how likely that assumption was.
Why did my result become significant when I collected more data?
Because the test measures how well you measured, not how big the difference is. The standard error shrinks with the square root of the number of observations, so the same difference produces a bigger statistic every time the sample grows.
A difference of a quarter of a standard deviation gives 0.4495 on ten observations, 0.0576 on sixty, 0.0141 on a hundred and 0.0005 on two hundred. The difference never changed once. That is why the size of the difference, counted in standard deviations, sits next to the answer on this page.
| Observations | p value | Significant at 5% |
|---|---|---|
| 10 | 0.449524 | no |
| 30 | 0.181415 | no |
| 60 | 0.057602 | no |
| 100 | 0.014063 | yes |
| 200 | 0.000506 | yes |
| 500 | 0.000000 | yes |
Can a significant result still be unimportant?
Yes, and it happens constantly in large samples. Significant means the difference is hard to explain by chance, not that it is large enough to act on.
A drug that lowers blood pressure by a hundredth of a point will be significant if you test it on enough people, and useless either way. Read the size of the difference first and the p value second.
Which significance level should I pick?
5% is the convention in most fields, but it is a convention and not a law, and it should be chosen before you see the data rather than after.
Lower it when a false alarm is expensive, as in a decision that is hard to undo. Raise it when missing a real effect is the worse mistake, as in a cheap screening step that will be checked again later.
What does a one sided test change?
It puts the whole tolerance on one side, so it needs less evidence to declare a difference in the direction you named, and none at all in the other direction.
That only works if you name the direction before looking. Choosing to test only for a higher value because the average happened to come out higher is the oldest way there is to turn a coin flip into a finding, and it doubles the false alarm rate you think you are running.
Reactions
0
0 Comments
Be the first to comment