Answer

PCA, eigenvectors and eigenvalues explained in plain language

Inspired by a question on Cross Validated · September 27, 2026

pcanormalization

The idea without the algebra

Picture a cloud of points: every person in a study measured on two variables, such as height and weight. The cloud is not round. It is stretched along a diagonal, because tall people tend to be heavier. If you could only describe each person with one number, the most informative single number would be their position along that long diagonal, roughly 'overall size'. Their position across the diagonal, 'heavier or lighter than expected for their height', is the second, smaller piece of information.

PCA does exactly this, for any number of variables. It finds the direction in which the cloud is longest and calls it the first principal component. Then it finds the longest direction at right angles to the first, and so on, until there are as many components as variables. Nothing is lost at that point: it is the same cloud seen from a better angle. The gain comes when you notice that the last few directions are thin, so you can drop them and keep most of the information in far fewer numbers.

Where eigenvectors and eigenvalues come in

The shape of the cloud is summarized by its covariance matrix, or by its correlation matrix if the variables are standardized first. A matrix acts on directions: push a direction through it and it usually comes out pointing somewhere else. An eigenvector is a special direction that comes out pointing the same way, only stretched. The eigenvalue is how much it gets stretched.

For a covariance or correlation matrix, those special directions are the axes of the cloud, and the stretch factor equals the variance of the data along that axis. So:

• Eigenvector = the recipe for a component: a set of weights, one per original variable, that defines the new axis.
• Eigenvalue = how much of the total variance lies along that axis. Dividing each eigenvalue by the sum of all of them gives the proportion of variance explained.
• The eigenvectors are at right angles to each other, which is why the component scores are uncorrelated.

With standardized variables each variable contributes a variance of 1, so the eigenvalues add up to the number of variables. An eigenvalue above 1 means the component carries more variance than any single original variable.

See it in R and Python

This simulation creates six test scores for 300 people. Three depend on a verbal ability and three on a numeric ability, and the two abilities are correlated. PCA is done twice in R: by hand with eigen() on the correlation matrix, and with base R's prcomp(). The Python tab does the same with NumPy.

set.seed(2691)
n <- 300
# Two related hidden abilities drive six test scores
verbal  <- rnorm(n)
numeric <- 0.5 * verbal + rnorm(n, sd = 0.87)
X <- data.frame(
  vocab   = verbal  + rnorm(n, sd = 0.5),
  reading = verbal  + rnorm(n, sd = 0.5),
  writing = verbal  + rnorm(n, sd = 0.6),
  algebra = numeric + rnorm(n, sd = 0.5),
  geometry= numeric + rnorm(n, sd = 0.5),
  stats   = numeric + rnorm(n, sd = 0.6)
)

# PCA by hand: eigen-decomposition of the correlation matrix
e <- eigen(cor(X))
round(e$values, 3)                       # eigenvalues = variance of each component
round(e$values / sum(e$values), 3)       # proportion of variance explained
round(cumsum(e$values) / ncol(X), 3)     # cumulative proportion

# The same thing with prcomp (standardized variables)
p <- prcomp(X, scale. = TRUE)
round(p$sdev^2, 3)                       # identical eigenvalues
round(p$rotation[, 1:2], 3)              # eigenvectors = loadings (signs are arbitrary)

# Scores: each person's position along the new axes
scores <- scale(X) %*% e$vectors
round(apply(scores, 2, var), 3)          # variances equal the eigenvalues
round(cor(scores[, 1], scores[, 2]), 3)  # components are uncorrelated

import numpy as np

rng = np.random.default_rng(2691)
n = 300
# Two related hidden abilities drive six test scores
verbal = rng.normal(0, 1, n)
numeric = 0.5 * verbal + rng.normal(0, 0.87, n)
X = np.column_stack([
    verbal + rng.normal(0, 0.5, n),    # vocab
    verbal + rng.normal(0, 0.5, n),    # reading
    verbal + rng.normal(0, 0.6, n),    # writing
    numeric + rng.normal(0, 0.5, n),   # algebra
    numeric + rng.normal(0, 0.5, n),   # geometry
    numeric + rng.normal(0, 0.6, n),   # stats
])

# PCA by hand: eigen-decomposition of the correlation matrix
values, vectors = np.linalg.eigh(np.corrcoef(X, rowvar=False))
order = np.argsort(values)[::-1]              # eigh sorts ascending; flip to largest first
values, vectors = values[order], vectors[:, order]
print(np.round(values, 3))                    # eigenvalues = variance of each component
print(np.round(values / values.sum(), 3))     # proportion of variance explained
print(np.round(np.cumsum(values) / X.shape[1], 3))
print(np.round(vectors[:, :2], 3))            # eigenvectors = loadings (signs are arbitrary)

# Scores: each person's position along the new axes
Z = (X - X.mean(axis=0)) / X.std(axis=0, ddof=1)
scores = Z @ vectors
print(np.round(scores.var(axis=0, ddof=1), 3))   # variances equal the eigenvalues
print(round(np.corrcoef(scores[:, 0], scores[:, 1])[0, 1], 3))  # uncorrelated

The figures below come from one seeded run of the R code. Python uses a different random number generator, so its figures differ slightly, but the pattern is the same (its eigenvectors may also come out with the opposite sign, which changes nothing).

• Two big eigenvalues, four small ones. The eigenvalues are 3.520, 1.557, 0.315, 0.234, 0.199 and 0.176. They add up to 6, the number of standardized variables.
• Most of the information fits in two numbers. The first component explains 58.7% of the variance and the second 25.9%, together 84.6%. The remaining four components are mostly measurement noise.
• The eigenvectors say what each component means. The first has similar positive weights on all six tests (0.379 to 0.428): it is general performance. The second has positive weights on the three verbal tests (0.380 to 0.403) and negative weights on the three numeric ones (-0.408 to -0.439): it contrasts verbal with numeric strength.
• Eigenvalues really are variances. The variances of the component scores equal the eigenvalues exactly, prcomp() gives the same eigenvalues as eigen(), and the first two component scores have a correlation of 0.

Practical points

• Scale first when units differ. PCA on raw variables is driven by whichever variable has the largest variance, so income in dollars would swamp age in years. Using the correlation matrix, or prcomp(x, scale. = TRUE), gives every variable equal weight. See centering vs standardizing for more on scaling.
• Signs are arbitrary. An eigenvector multiplied by -1 is still an eigenvector, so different programs can flip a component. Interpret the pattern of weights, not the overall sign.
• Choosing how many components to keep. Common guides are a scree plot (keep components before the curve flattens), eigenvalues above 1 for standardized data, and a target cumulative percentage. Parallel analysis, which compares your eigenvalues with those from random data, is usually the most defensible choice.
• PCA is not factor analysis. PCA summarizes all the variance, including measurement error, into components. Factor analysis models only the shared variance as coming from latent factors. Use PCA to compress or describe data, and factor analysis when you want to measure an underlying construct.
• Weights versus loadings. prcomp() reports eigenvector weights. Many programs instead report loadings scaled by the square root of the eigenvalue, which equal the correlations between variables and components. Say which you are showing.

How to report a principal component analysis in APA style (7th edition)

In the method section, say which variables were entered, whether PCA used the correlation or covariance matrix, whether any rotation was applied, and how you decided how many components to keep. Using the example above:

"A principal component analysis of the six standardized test scores (N = 300) yielded two components with eigenvalues greater than 1. The first component (eigenvalue = 3.52) explained 58.7% of the variance and had similar positive weights on all tests; the second (eigenvalue = 1.56) explained a further 25.9% and contrasted verbal with numeric tests. Together the two components explained 84.6% of the variance."

Put the full set of weights or loadings in a table, with one row per variable, one column per retained component, and rows at the bottom for the eigenvalue and percentage of variance. State in a table note whether the values are eigenvector weights or loadings.

Related tools and guides

• Which statistical test should I use?
• Centering vs standardizing predictors
• Principal component analysis (Wikipedia)
• Eigenvalues and eigenvectors (Wikipedia)

More answered questions

• Centering vs standardizing predictors in regression: when to do which?
• How do you normalize data to a 0-1 range?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.