Tag
hypothesis testing
28 answered questions tagged “hypothesis testing”. Browse and search all answers, or ask your own.
Searches the hypothesis testing answers on this page. Search all answers
Bayesian vs frequentist statistics in plain English
Frequentists treat probability as long-run frequency; Bayesians treat it as degree of belief. Here is what that changes, with one example done both ways.
Read the answer →Can you run a chi-square test on weighted survey estimates?
A chi-square test on weighted survey totals treats millions of estimated people as real observations. Why that fails and what to do instead.
Read the answer →Chi-square goodness of fit vs test of independence: what is the difference?
Both tests use the same Pearson formula. What differs is the question, where the expected counts come from, and so the degrees of freedom.
Read the answer →Why does a t test use the t distribution and ANOVA the F distribution?
How the t, F and chi-square reference distributions are derived from the way each test statistic is built, and what to do when no formula exists.
Read the answer →Chi-square test with expected counts below 5: what should you do?
What the rule of 5 really checks, when sparse cells matter, and how Fisher's exact test or a Monte Carlo p value fixes 2x2 and larger tables.
Read the answer →How do you test whether event counts vary by month?
A chi-square goodness-of-fit test weighted by month length, its blind spots, and a Poisson regression that tests one seasonal wave on 2 df.
Read the answer →How do you compare medians when the groups differ in shape and spread?
Why Mann-Whitney is not a test of medians when shapes differ, and how Mood's median test and a bootstrap interval compare medians directly.
Read the answer →Kolmogorov-Smirnov, t test or Mann-Whitney: which detects a change from a baseline?
What each two-sample test detects, why Kolmogorov-Smirnov misses changes in spread with small samples, and how to compare against a baseline.
Read the answer →Significant ANOVA but no significant pairwise comparisons: how can that happen?
Why a one-way ANOVA F test can reject while every pairwise or Tukey comparison does not (and the reverse), and how far apart their p values can get.
Read the answer →t tests vs the F test in regression: what is the difference?
Why regression output gives a t test for each coefficient and one overall F test, when the two agree exactly, and why they can point in opposite directions.
Read the answer →Why does the Welch t statistic only approximately follow a t distribution?
Why a sum of two weighted chi-square variances breaks the exact t distribution, and how the Satterthwaite fractional df approximate it.
Read the answer →False positive risk: how likely is a significant finding to be wrong?
Why low power raises the share of significant findings that are false, inflates the real effects that get through, and what one researcher can do.
Read the answer →What should you do after rejecting the null hypothesis?
A significant result is a starting point, not a conclusion. Estimate the effect, check that it is robust, and replicate before you treat it as established.
Read the answer →How to interpret the F statistic and p value in a one-way ANOVA
What the F ratio in an ANOVA table measures, how it maps onto the p value and the critical F, and what a significant F does and does not tell you.
Read the answer →Planned contrasts vs post hoc tests: why do they give different p-values?
Why one comparison gets different p-values as a planned contrast, a Tukey test or a Bonferroni test, and when to use each.
Read the answer →Bonferroni correction: when should you use it, and what are the alternatives?
What a Bonferroni correction protects against, why it costs so much power, and how Holm and Benjamini-Hochberg compare.
Read the answer →Why does almost everything become statistically significant with a large sample?
Large samples do not bias a test toward rejecting a true null; they detect tiny real differences. How to tell significant from important.
Read the answer →Paired vs independent t test: which should you use?
How to tell whether your data are paired, why pairing can turn a hopeless comparison into a clear one, and what goes wrong when you pick the wrong test.
Read the answer →How are alpha and beta (Type I and Type II error rates) related?
Why lowering alpha raises beta, why the trade-off is not linear or a fixed ratio, and what a Bonferroni correction does to power.
Read the answer →Is the chi-square test one-tailed or two-tailed?
Why a chi-square test reads only the right end of its curve yet detects differences in any direction, and why its p value is never halved.
Read the answer →Is a non-significant result from a large study evidence for the null?
When failing to reject the null hypothesis tells you something, why the p value alone cannot show it, and how equivalence tests make the case.
Read the answer →Welch vs Student t test: should you just always use Welch?
Why Welch's t test is the safer default for two independent groups, what it costs when variances are equal, and where Mann-Whitney fits.
Read the answer →One-tailed vs two-tailed tests: why not just test the direction the data point to?
Why a two-tailed test is the default, why choosing the tail after seeing the data doubles your false-positive rate, and when a one-tailed test is legitimate.
Read the answer →What do p values and t values actually mean?
A t value is an effect measured in standard errors; the p value turns it into a tail probability under the null. What both mean, in plain language.
Read the answer →What are the assumptions of the Wilcoxon signed-rank test?
Independent pairs, rankable differences and symmetry: what each assumption means, and why a non-significant symmetry test proves little.
Read the answer →What are degrees of freedom in statistics, really?
Degrees of freedom count how much of your data is left for estimating variability once other quantities are estimated. The idea in plain language.
Read the answer →Is there a 95% probability that your confidence interval covers the true mean?
The 95% describes the method that builds the interval, not the one interval you computed. Here is why that distinction matters, and when it does not.
Read the answer →Is a normality test worth running on your data?
Normality tests answer the wrong question: with lots of data they flag harmless quirks, with little data they miss real problems. Here is what to do.
Read the answer →