Answer

Can you run a chi-square test on weighted survey estimates?

Inspired by a question on Cross Validated · October 28, 2026

chi-squarehypothesis testingsamplingeffect size

The short answer

Official statistics such as labour-force or census-style tables usually come from a survey: tens of thousands of people are interviewed, and each one is given a weight saying how many people in the population they stand for. The published table then shows estimated totals in thousands or millions. Those numbers are estimates of the population, not counts of observations.

A Pearson chi-square test assumes each cell holds a count of independently sampled cases. Give it 139 million estimated people and it believes you observed 139 million people, so the standard errors it implies are absurdly small and almost any gap comes out with a microscopic p-value. The test is answering a question about a sample that never existed.

The real uncertainty depends on how many people were actually interviewed and how the survey was designed (clustering, stratification, unequal weights). So the right comparison uses the standard errors of the two estimated proportions, which statistical agencies usually publish or explain how to compute.

Why weighted totals break the test

The clearest symptom is that the answer depends on the units. Multiply every cell of a table by a constant k and the proportions do not change at all, but the Pearson statistic is multiplied by exactly k. A table reported in thousands and the same table reported in persons describe the same population, yet the second gives a chi-square 1,000 times larger. A valid test cannot change its verdict because you moved a decimal point.

• Weights are not extra information. A respondent with a weight of 22,000 is still one interview. Weighting corrects who is represented; it does not create new data.
• Complex designs add variance. Interviewing households in selected areas, rather than people chosen completely at random, usually makes estimates less precise than a simple random sample of the same size. The design effect (deff) is that ratio of variances, and the effective sample size is roughly n / deff.
• Stitching together confidence limits is not a test. Taking the end of one interval that shrinks the gap and the opposite end of the other is close to asking whether two intervals overlap. That check is overly conservative (two 95% intervals can overlap even when the difference is significant), and it ignores that the totals and their denominators are estimated together.

See it in R and Python

The code imagines a survey of 6,000 people (3,100 men and 2,900 women) weighted up to a population of 70 million men and 65 million women. In the sample, 40% of men and 43% of women hold a degree. It runs the chi-square test the wrong way, on the weighted totals, and then the right way, on the people interviewed, first as if the survey were a simple random sample and then with an assumed design effect of 1.5. Finally it simulates 2,000 surveys in which men and women truly have the same rate (41%) and counts how often each approach declares a significant difference.

set.seed(331972)

# A survey of 6,000 people stands in for 135 million (70 million men, 65 million women)
n   <- c(men = 3100, women = 2900)        # people actually interviewed
deg <- c(men = 1240, women = 1247)        # of whom hold a degree (40% vs 43%)
w   <- c(men = 70e6, women = 65e6) / n    # survey weight: people each respondent represents

# 1. WRONG: chi-square on the weighted population estimates
est <- cbind(degree = deg * w, no_degree = (n - deg) * w)
naive_thousands <- chisq.test(est / 1000, correct = FALSE)$statistic
naive_persons   <- chisq.test(est, correct = FALSE)$statistic
round(c(thousands = unname(naive_thousands), persons = unname(naive_persons),
        ratio = unname(naive_persons / naive_thousands)), 1)

# 2. RIGHT (simple random sample): test on the people actually interviewed
pt <- prop.test(deg, n, correct = FALSE)
round(c(chisq = unname(pt$statistic), p = pt$p.value), 4)
p_hat <- deg / n
round(c(p_hat, diff = unname(p_hat[2] - p_hat[1]), ci = -rev(pt$conf.int),
        ratio = unname(p_hat[1] / p_hat[2])), 4)

# 3. With a design effect (clustering, unequal weights) the SEs are larger
deff <- 1.5
se <- sqrt(p_hat * (1 - p_hat) / n * deff)
se_diff <- sqrt(sum(se^2))
z <- unname(p_hat[2] - p_hat[1]) / se_diff
round(c(se_diff = se_diff, z = z, p = 2 * pnorm(-abs(z)),
        lo = unname(p_hat[2] - p_hat[1]) - 1.96 * se_diff,
        hi = unname(p_hat[2] - p_hat[1]) + 1.96 * se_diff), 4)

# 4. When the two rates are truly equal, how often does each approach reject?
sims <- replicate(2000, {
  d <- rbinom(2, n, 0.41)
  tab <- cbind(d, n - d)
  c(naive = chisq.test(tab * w, correct = FALSE)$p.value < 0.05,
    right = chisq.test(tab, correct = FALSE)$p.value < 0.05)
})
rowMeans(sims)

import numpy as np
from scipy import stats

rng = np.random.default_rng(331972)

# A survey of 6,000 people stands in for 135 million (70 million men, 65 million women)
n = np.array([3100, 2900])                # people actually interviewed (men, women)
deg = np.array([1240, 1247])              # of whom hold a degree (40% vs 43%)
w = np.array([70e6, 65e6]) / n            # survey weight: people each respondent represents

# 1. WRONG: chi-square on the weighted population estimates
est = np.column_stack([deg * w, (n - deg) * w])
naive_thousands = stats.chi2_contingency(est / 1000, correction=False)[0]
naive_persons = stats.chi2_contingency(est, correction=False)[0]
print({"thousands": round(naive_thousands, 1), "persons": round(naive_persons, 1),
       "ratio": round(naive_persons / naive_thousands, 1)})

# 2. RIGHT (simple random sample): test on the people actually interviewed
tab = np.column_stack([deg, n - deg])
chi2, p = stats.chi2_contingency(tab, correction=False)[:2]
print({"chisq": round(chi2, 4), "p": round(p, 4)})
p_hat = deg / n
diff = p_hat[1] - p_hat[0]
se_srs = np.sqrt(np.sum(p_hat * (1 - p_hat) / n))   # unpooled, as in R's prop.test CI
z975 = stats.norm.ppf(0.975)
print({"men": round(p_hat[0], 4), "women": round(p_hat[1], 4), "diff": round(diff, 4),
       "ci1": round(diff - z975 * se_srs, 4), "ci2": round(diff + z975 * se_srs, 4),
       "ratio": round(p_hat[0] / p_hat[1], 4)})

# 3. With a design effect (clustering, unequal weights) the SEs are larger
deff = 1.5
se = np.sqrt(p_hat * (1 - p_hat) / n * deff)
se_diff = np.sqrt(np.sum(se ** 2))
z = diff / se_diff
print({"se_diff": round(se_diff, 4), "z": round(z, 4), "p": round(2 * stats.norm.sf(abs(z)), 4),
       "lo": round(diff - 1.96 * se_diff, 4), "hi": round(diff + 1.96 * se_diff, 4)})

# 4. When the two rates are truly equal, how often does each approach reject?
naive, right = [], []
for _ in range(2000):
    d = rng.binomial(n, 0.41)
    t = np.column_stack([d, n - d])
    naive.append(stats.chi2_contingency(t * w[:, None], correction=False)[1] < 0.05)
    right.append(stats.chi2_contingency(t, correction=False)[1] < 0.05)
print({"naive": np.mean(naive), "right": np.mean(right)})

The figures below come from one seeded run of the R code. Parts 1 to 3 involve no randomness, and the Python version reproduces them exactly. Python's random numbers differ from R's, so its simulation gives slightly different rejection rates (99.15% and 4.45%), but the same pattern.

• The naive test depends on the units. On the weighted totals in thousands, the chi-square statistic is 125.0; on the same totals in persons it is 124,993, exactly 1,000 times larger. Neither number means anything.
• Tested on the 6,000 interviews as a simple random sample, the gap of 3.0 percentage points (40.0% vs 43.0%) gives χ² = 5.56, p = .018, with a 95% CI for the difference of 0.51 to 5.49 points. Men were 0.93 times as likely as women to hold a degree.
• With a design effect of 1.5, the standard error of the difference grows to 0.0156, z = 1.92, p = .054, and the 95% CI runs from -0.05 to 6.05 points. The same data are now borderline, which is why the design matters.
• Under a true null, the naive test on weighted totals declared a significant difference in 99.3% of the 2,000 simulated surveys. The test on the actual interviews did so 4.25% of the time, close to the promised 5%.

What to do instead

• Use the published standard errors. If the agency gives a standard error (or a margin of error, which is usually 1.96 standard errors) for each proportion, the standard error of the difference is roughly √(SE₁² + SE₂²), and z = difference / SE. This ignores any covariance between the two estimates, an approximation many agencies' own guidance also uses.
• Work with proportions, not totals. Compare the rate in each group. The intervals for the counts and for the group sizes do not combine into an interval for the rate by picking their extreme ends.
• With the microdata, use design-based methods. Software that knows the weights, strata and clusters computes correct standard errors; for a full contingency table, the Rao-Scott corrected chi-square (for example svychisq() in R's survey package) replaces the ordinary test.
• Report the size of the gap, not only the p-value. A difference in percentage points and a ratio of rates, each with a confidence interval, show whether it matters. Large surveys often pin down even small gaps; see why large samples make tiny effects significant.

If you truly have every member of a population (a full register rather than a survey), there is no sampling error to test, and the observed difference is simply the difference. A test then only makes sense if you are asking about an underlying process that could have produced other populations. For choosing a test when you do have a sample, see the test chooser, and for how the chi-square test of independence works, see goodness of fit vs test of independence.

How to report a difference in proportions in APA style (7th edition)

Give both percentages, the difference with its confidence interval, the test statistic and the p-value, and say in the method section how the standard errors account for the survey design. Using the design-based example above:

"Using survey weights and an assumed design effect of 1.5, an estimated 43.0% of women and 40.0% of men held a degree, a difference of 3.0 percentage points, 95% CI [-0.05, 6.05], z = 1.92, p = .054."

If the data were a simple random sample, a chi-square test on the unweighted counts is fine: "Degree holding differed by sex, χ²(1, N = 6,000) = 5.56, p = .018; 43.0% of women and 40.0% of men held a degree (difference = 3.0 points, 95% CI [0.51, 5.49])." Never report N as the weighted population total. For more on reporting survey results, see the DASS blog.

Related tools and guides

• Chi-square power and sample size calculator
• APA 7 formatter for chi-square tests
• Check a reported p-value
• Chi-square goodness of fit vs test of independence
• Why does almost everything become significant with a large sample?
• Where does the 1/√n margin of error come from?
• Which statistical test should I use?
• Design effect (Wikipedia)

More answered questions

• Chi-square goodness of fit vs test of independence: what is the difference?
• Chi-square test with expected counts below 5: what should you do?
• How do you test whether event counts vary by month?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.