AskStats

Answer

What do p values and t values actually mean?

Inspired by a question on Cross Validated ·

p-valuest testhypothesis testing

The short answer

A t value is a signal-to-noise ratio: the estimated effect divided by its standard error. The p value is the probability of getting a t at least that far from zero if the null hypothesis were true, so a small p says the data would be surprising under the null. It is not the probability that the null is true, and a large p value never proves that there is no effect.

The short version

Every hypothesis test starts from a null hypothesis, usually the boring claim that there is no difference or no effect. The test then asks one question: if the null were true, how unusual would data like ours be?

The t value answers the first half. It measures how far the estimate is from what the null predicts, in units of standard error. A t of 0.5 means the estimate is half a standard error from zero, well within ordinary sampling noise. A t of 3 means it is three standard errors away, which chance alone rarely produces.

The p value turns that distance into a probability. It is the chance, computed as if the null were true, of getting a t at least as far from zero as the one you observed. Small p values mean your result would be rare under the null, which counts as evidence against it.

How the t value and the p value are connected

For a comparison of two means, t is the difference between the sample means divided by the standard error of that difference. The numerator is the signal; the denominator is how much that signal would bounce around from sample to sample. More data or less variable data shrink the standard error, so the same difference produces a larger t.

If the null is true and the assumptions hold, t follows a t distribution with a known number of degrees of freedom. The two-sided p value is simply the area in both tails of that distribution beyond the observed t. So p is a one-to-one translation of t, given the df: bigger |t|, smaller p.

The df matter. With few df the t distribution has heavy tails, so the same t is less surprising. In the example below, t = 2.1 gives p = .090 with 5 df, p = .049 with 20 df and p = .038 with 100 df.

See it in R and Python

The code compares two simulated groups of 20 whose true means differ by 6 points (SD 10), computes t and p by hand to show they match t.test(), and then repeats the experiment 10,000 times with and without a real difference to show how p values behave.

R

set.seed(31)

# 1. Two groups of 20; the true difference in means is 6 points (SD 10)
a <- rnorm(20, mean = 56, sd = 10)
b <- rnorm(20, mean = 50, sd = 10)
tt <- t.test(a, b, var.equal = TRUE)

# The t value is the difference divided by its standard error
diff <- mean(a) - mean(b)
sp   <- sqrt(((20 - 1) * var(a) + (20 - 1) * var(b)) / (20 + 20 - 2))
se   <- sp * sqrt(1 / 20 + 1 / 20)
round(c(diff = diff, se = se, t_by_hand = diff / se,
        t_from_test = unname(tt$statistic), df = unname(tt$parameter)), 3)

# The p value is the tail area of the t distribution beyond |t|, both sides
round(c(p_by_hand = 2 * pt(-abs(diff / se), df = 38), p_from_test = tt$p.value), 4)
round(c(mean_a = mean(a), sd_a = sd(a), mean_b = mean(b), sd_b = sd(b)), 2)
round(tt$conf.int, 2)

# 2. The same t value gives different p values with different df
round(sapply(c(df5 = 5, df20 = 20, df100 = 100), function(k) 2 * pt(-2.1, k)), 4)

# 3. When the null is true, p values are spread evenly between 0 and 1
p_null <- replicate(10000, t.test(rnorm(20, 50, 10), rnorm(20, 50, 10),
                                  var.equal = TRUE)$p.value)
round(mean(p_null < 0.05), 3)                       # about 0.05
round(table(cut(p_null, seq(0, 1, 0.2))) / 10000, 3) # about 0.2 in each bin

# 4. When there is a real 6-point difference, small p values pile up
p_alt <- replicate(10000, t.test(rnorm(20, 56, 10), rnorm(20, 50, 10),
                                 var.equal = TRUE)$p.value)
round(mean(p_alt < 0.05), 3)                        # power of this design
round(mean(p_alt > 0.5), 3)                         # a large p is still common

Python

import numpy as np
from scipy import stats

rng = np.random.default_rng(31)

# 1. Two groups of 20; the true difference in means is 6 points (SD 10)
a = rng.normal(56, 10, 20)
b = rng.normal(50, 10, 20)
tt = stats.ttest_ind(a, b)  # equal variances assumed, like var.equal = TRUE

# The t value is the difference divided by its standard error
diff = a.mean() - b.mean()
sp = np.sqrt(((20 - 1) * a.var(ddof=1) + (20 - 1) * b.var(ddof=1)) / (20 + 20 - 2))
se = sp * np.sqrt(1 / 20 + 1 / 20)
print("diff", round(diff, 3), "se", round(se, 3), "t_by_hand", round(diff / se, 3),
      "t_from_test", round(tt.statistic, 3), "df", tt.df)

# The p value is the tail area of the t distribution beyond |t|, both sides
print("p_by_hand", round(2 * stats.t.cdf(-abs(diff / se), 38), 4), "p_from_test", round(tt.pvalue, 4))
ci = tt.confidence_interval(0.95)
print("95% CI", round(ci.low, 2), round(ci.high, 2))

# 2. The same t value gives different p values with different df
print({k: round(2 * stats.t.cdf(-2.1, k), 4) for k in [5, 20, 100]})

# 3. When the null is true, p values are spread evenly between 0 and 1
p_null = stats.ttest_ind(rng.normal(50, 10, (10000, 20)),
                         rng.normal(50, 10, (10000, 20)), axis=1).pvalue
print(round(np.mean(p_null < 0.05), 3))                       # about 0.05
print(np.round(np.histogram(p_null, bins=np.linspace(0, 1, 6))[0] / 10000, 3))

# 4. When there is a real 6-point difference, small p values pile up
p_alt = stats.ttest_ind(rng.normal(56, 10, (10000, 20)),
                        rng.normal(50, 10, (10000, 20)), axis=1).pvalue
print(round(np.mean(p_alt < 0.05), 3))                        # power of this design
print(round(np.mean(p_alt > 0.5), 3))                         # a large p is still common

The figures below come from one seeded run of the R code. Python uses a different random number generator, so its simulated figures differ slightly, but they show the same pattern; the df comparison, which involves no randomness, matches exactly.

Should you hope for a high or a low p value?

A low p value is what you want when your goal is to show that an effect exists, because it is evidence against the no-effect null. People often hope for a high p value when checking an assumption, such as a normality test or a goodness-of-fit test, since there the null is the thing they want to be true.

That second use is weaker than it looks. A high p value means the data are compatible with the null, not that the null has been confirmed. The simulation above shows why: a real difference produced p > .5 about one time in ten. If you need to show that two groups are practically the same, use an equivalence test (such as two one-sided tests) with a pre-set margin, or report a confidence interval and show it contains only negligible differences.

Common misreadings

The American Statistical Association's 2016 statement on p-values makes these same points (Wasserstein & Lazar, 2016). For the role of df in the t distribution, see What are degrees of freedom in statistics, really?. For more plain-language guides to planning and reporting analyses, see the DASS blog.

How to report a t test and p value in APA style (7th edition)

Give the t value with its df in parentheses, the exact p value to two or three decimals without a leading zero, and the confidence interval for the effect. Using the example above:

"Group A scored higher (M = 55.26, SD = 8.83) than group B (M = 48.02, SD = 9.88), t(38) = 2.45, p = .019, 95% CI [1.25, 13.24]."

Report exact p values rather than "p < .05" or "n.s."; for very small values write p < .001. Add an effect size such as Cohen's d (d = x.xx) when you can. In the method section, say which test you used and whether equal variances were assumed.

Related tools and guides

More answered questions

Working with your own data?

General answers only go so far. Send us your situation and we'll reply by email within two business days.

Ask your question

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.