Why Do We Divide by n-1 for Sample Variance? (The Intuitive & Mathematical Guide)


Why Do We Divide by (n - 1) in Sample Variance?

Key point: Sample variance uses (n - 1) to remove bias caused by estimating the mean.

Sample variance is defined as:

s2 = (1 / (n - 1)) ∑i=1n (xi - x̄)2

The use of (n - 1) ensures that the estimator is unbiased.

1. Problem Definition

Let a population have mean μ and variance σ2. We draw an i.i.d. sample:

x1, ..., xn

Sample mean:

x̄ = (1 / n) ∑ xi

2. Naive Estimator and Bias

Naive estimator:

(1 / n) ∑(xi - x̄)2

This estimator is biased downward:

E[~s2] < σ2

3. Sample Mean Reduces Spread (Visualization)

The sample mean minimizes squared deviations, which reduces observed spread.

Figure 1. Sample points with true mean and sample mean. The sample mean shifts toward the data, reducing apparent variance.

4. Mathematical Derivation

We now derive why dividing by n - 1 makes the sample variance unbiased.

Assume that x1, x2, ..., xn are i.i.d. random samples from a population with mean μ and variance σ2. Define the sample mean as

x̄ = (1 / n) × (x1 + x2 + ... + xn)

We want to compute the expectation of

(x1 - x̄)2 + (x2 - x̄)2 + ... + (xn - x̄)2

because this is the numerator of the sample variance.

4.1 Expand Around the True Mean

Start from the identity

xi - x̄ = (xi - μ) - (x̄ - μ)

Then square both sides:

(xi - x̄)2 = [(xi - μ) - (x̄ - μ)]2

Using (a - b)2 = a2 - 2ab + b2,

(xi - x̄)2 = (xi - μ)2 - 2(xi - μ)(x̄ - μ) + (x̄ - μ)2

Now sum over all i:

∑(xi - x̄)2 = ∑(xi - μ)2 - 2(x̄ - μ)∑(xi - μ) + ∑(x̄ - μ)2

Since (x̄ - μ) does not depend on i,

∑(x̄ - μ)2 = n(x̄ - μ)2

Also,

∑(xi - μ) = (x1 + x2 + ... + xn) - nμ = n x̄ - nμ = n(x̄ - μ)

Substituting this result,

∑(xi - x̄)2 = ∑(xi - μ)2 - 2n(x̄ - μ)2 + n(x̄ - μ)2

Therefore,

∑(xi - x̄)2 = ∑(xi - μ)2 - n(x̄ - μ)2

4.2 Take Expectation

Take expectation on both sides:

E[∑(xi - x̄)2] = E[∑(xi - μ)2] - E[n(x̄ - μ)2]

4.3 First Term

Because each sample has variance σ2,

E[(xi - μ)2] = σ2

So,

E[∑(xi - μ)2] = nσ2

4.4 Second Term

Since x̄ is the sample mean,

x̄ = (1 / n) × ∑xi

and E[x̄] = μ, we have

E[(x̄ - μ)2] = Var(x̄)

For i.i.d. samples,

Var(x̄) = σ2 / n

Hence,

E[n(x̄ - μ)2] = n × (σ2 / n) = σ2

4.5 Final Result

Substitute the two terms:

E[∑(xi - x̄)2] = nσ2 - σ2 = (n - 1)σ2

Now divide by (n - 1):

E[(1 / (n - 1)) × ∑(xi - x̄)2] = σ2

Therefore,

s2 = (1 / (n - 1)) × ∑(xi - x̄)2

is an unbiased estimator of the population variance.

4.6 Interpretation

If we divide by n instead, then

E[(1 / n) × ∑(xi - x̄)2] = ((n - 1) / n)σ2

which is smaller than σ2. Therefore, the estimator with denominator n is biased downward.

5. Simulation: Bias Comparison

Distribution:

X ~ N(0, 1)

Comparison of estimators:

  • Divide by n → biased
  • Divide by n - 1 → unbiased

Figure 2. Histogram of variance estimates. The n-based estimator is biased downward, while the (n-1) estimator is centered at the true variance.

6. Effect of Sample Size

Bias decreases as sample size increases.

Figure 3. Bias vs sample size. The estimator dividing by n remains biased, while (n-1) converges to zero bias.

7. Degrees of Freedom

Constraint:

∑(xi - x̄) = 0

Only n - 1 independent deviations exist.

8. Practical Impact

  • Small n → significant difference
  • Large n → negligible difference

n / (n - 1) → 1

Conclusion

Dividing by n - 1 corrects the bias introduced by estimating the mean from the same data. This ensures that the sample variance is an unbiased estimator of the population variance.

Comments

Post a Comment

Popular posts from this blog

Modal Analysis Part2. Windows

Image
In this article, we'll briefly explore the significance of windows in signal processing and provide a simple Python code example to create a Hanning window. Understanding Windows in Signal Processing In the realm of signal processing, windows play a pivotal role in preparing data for analysis in the frequency domain. A windowing function modifies a time-domain signal's properties before transforming it into the frequency domain through techniques like the Fast Fourier Transform (FFT).  What are Windows in Signal Processing? Windows, in signal processing, are mathematical functions applied to a dataset to mitigate issues like spectral leakage and artifacts during frequency analysis. These issues often arise due to abrupt signal endings, causing unwanted effects in the frequency domain. Windows act as a weighting function, tapering the signal's edges to minimize these problems. 1. Rectangular Window The Rectangular window, sometimes referred to as the boxcar or Dirichlet wind...
Read more

Averaging and Moving Average Filters

Image
  1. Why Averaging Matters In engineering systems, especially in robotics, sensor data is often noisy. A simple and effective way to reduce noise is to compute the average of measurements. However, how we compute the average matters, especially when data keeps growing. 2. Batch Expression The average of k data points is defined as: x̄ k = (x 1 + x 2 + ··· + x k ) / k This is called a batch expression , because all data must be collected before computing the result. 3. Limitation of the Batch Method The batch method has two major drawbacks: Every time new data arrives, we must recompute the entire sum. All previous data must be stored in memory. This becomes inefficient when the number of data points is large. 4. Recursive Averaging Filter Instead of recomputing everything, we can reuse the previous average. Starting from the batch expression: x̄ k = (x 1 + x 2 + ··· + x k ) / k Multiply both sides by k : k x̄ k = x 1 + x 2 + ··· + ...
Read more