Lesson 2.8: The Law of the Average: The WLLN

We now begin our journey into Asymptotic Theory—the study of what happens with large samples. This lesson introduces the Weak Law of Large Numbers (WLLN), the theorem that guarantees that with enough data, the sample average will inevitably converge to the true population average. This is the mathematical foundation for why backtesting and simulations work.

Part 1: The Certainty of Large Numbers

Why does a casino always make money? On any single spin of a roulette wheel, the outcome is random. But over millions of spins, the casino's average profit becomes a near-certainty. This phenomenon, where randomness averages out to a predictable constant, is guaranteed by the Law of Large Numbers.

The Core Idea: The WLLN formalizes the intuition that as you collect more data, your sample mean ( $\bar{X}$ ) gets closer and closer to the true population mean ( $\mu$ ).

To prove this, we first need a formal definition of "getting closer and closer." This is called convergence in probability.

Definition: Convergence in Probability

An estimator $\hat{\theta}_n$ converges in probability to a true value $\theta$ if the probability of the estimator being "far away" from the true value shrinks to zero as the sample size $n$ goes to infinity.

We write this as $\text{plim}(\hat{\theta}_n) = \theta$ , which formally means:

lim_{n \to infty} P(|\hat{ heta}_n - \theta| > \epsilon) = 0

Where $\epsilon$ is any small, positive number representing "far away."

Intuition: The Funnel of Certainty

Imagine the distribution of your sample mean $\bar{X}$ . When $n$ is small, the distribution is wide and uncertain. As $n$ grows, the distribution gets squeezed tighter and tighter around the true mean $\mu$ , like a funnel guiding your estimate to the correct value.

Part 2: The Weak Law of Large Numbers (WLLN)

The WLLN is the theorem that proves that the sample mean has this "funnel" property.

The Weak Law of Large Numbers (WLLN)

Let $X_1, \dots, X_n$ be a sequence of i.i.d. random variables with a finite mean $\mu$ and finite variance $\sigma^2$ . Then the sample mean, $\bar{X}_n$ , converges in probability to $\mu$ .

ar{X}_n \xrightarrow{p} mu quad ext{or} quad ext{plim}(ar{X}_n) = mu

The Proof Using Chebyshev's Inequality

The proof is surprisingly elegant and relies on a "brute-force" tool called Chebyshev's Inequality, which provides a loose upper bound on the probability of a random variable being far from its mean.

Proof of WLLN

Step 1: State Chebyshev's Inequality for $\bar{X}_n$ .

Chebyshev's states that for any $\epsilon > 0$ : $P(|\bar{X}_n - E[\bar{X}_n]| \ge \epsilon) \le \frac{\text{Var}(\bar{X}_n)}{\epsilon^2}$ .

Step 2: Substitute the known mean and variance of the sample mean.

We know $E[\bar{X}_n] = \mu$ and $\text{Var}(\bar{X}_n) = \sigma^2/n$ .

P(|\bar{X}_n - \mu| \ge \epsilon) \le \frac{\sigma^2/n}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}

Step 3: Take the limit as $n \to \infty$ .

We want to see what happens to this probability for a huge sample size.

\lim_{n \to \infty} P(|\bar{X}_n - \mu| \ge \epsilon) \le \lim_{n \to \infty} \frac{\sigma^2}{n\epsilon^2}

Since $\sigma^2$ and $\epsilon^2$ are fixed constants, the term on the right goes to zero as $n$ gets infinitely large.

\lim_{n \to \infty} \frac{\sigma^2}{n\epsilon^2} = 0

Conclusion: The probability of the sample mean being far from the true mean is squeezed between 0 and 0. Therefore, the limit must be 0. This is the definition of convergence in probability.

The Payoff: WLLN is the Foundation of Empirical Work

The WLLN is the bedrock that allows us to trust data. It is the formal link between theoretical probability and real-world observation.

It Justifies Backtesting: A quant who backtests a strategy over 5,000 trading days is calculating a sample average return. The WLLN is the mathematical guarantee that this number is a reliable (consistent) estimate of the strategy's true, long-run expected return. Without WLLN, backtesting would be a meaningless exercise.
It Powers Monte Carlo Simulations: When pricing a complex option, a quant might simulate 1,000,000 possible price paths and average the outcomes. The WLLN guarantees that as the number of simulations increases, this average will converge to the one true theoretical price.
It Guarantees Consistency: An estimator is **consistent** if it converges in probability to the true parameter. The WLLN is the simplest proof of consistency for the most common estimator, the sample mean. This property is the bare minimum requirement for any good estimator in econometrics or machine learning.

What's Next? The Shape of the Funnel

The WLLN is incredible. It tells us that the "funnel of certainty" for our sample mean narrows towards the true mean $\mu$ .

But it doesn't tell us the *shape* of that funnel. What is the *distribution* of the sample mean when $n$ is large? Is it skewed? Is it flat? Is it... Normal?

The next lesson introduces the most celebrated result in all of statistics—the **Central Limit Theorem (CLT)**—which gives us the stunning answer to this question.

Lesson 2.7: The F-Distribution (Fisher-Snedecor)

Lesson 2.9: The Universal Bell Curve: The Central Limit Theorem (CLT)