Lesson 1.3: Law of Total Probability and Bayes' Theorem

1.1 The Intuition

Before we can reverse conditional probability, we need a way to calculate the total, unconditional probability of an event $A$, given that $A$ can happen under several mutually exclusive scenarios.

Imagine an event: Event $A$ = My stock portfolio goes up tomorrow.

This can happen under two distinct, mutually exclusive scenarios:

1. Scenario $B$: The market is in a Bull Market (70% chance).
2. Scenario $B^c$: The market is in a Bear Market (30% chance).

The Law of Total Probability says the total probability of $A$ is the sum of the probabilities of $A$ happening in each scenario.

1.2 The General Case (Partitioning the Space)

If the sample space is partitioned into $k$ mutually exclusive events ($B_1, B_2, \dots, B_k$), the LTP is:

This allows us to calculate the marginal probability $P(A)$ using conditional probabilities, which is the entire denominator of Bayes' Theorem.

We now have the tools to solve the ultimate inference problem: reversing the condition.

• Given (Easy): We usually know the probability of the Data ($A$) given the Cause ($B$): $P(A | B)$. (E.g., probability of a test being positive given the patient has the disease).
• Desired (Hard): We want the probability of the Cause ($B$) given the Data ($A$): $P(B | A)$. (E.g., probability the patient has the disease given the test was positive).

Bayes' Theorem links these two using the General Multiplication Rule.

2.1 The Bayesian Terminology

Bayes' Theorem defines how we update our belief after observing data.

• $P(B)$: The Prior Probability (Our initial belief about the cause, $B$, before seeing the data $A$).
• $P(B | A)$: The Posterior Probability (Our updated belief about the cause, $B$, *after* seeing the data $A$).
• $P(A | B)$: The Likelihood (The probability of seeing the data $A$, given that our initial belief $B$ was correct).

3.1 The Machine Learning Connection: Naive Bayes and Classification

Bayes' Theorem is the fundamental theoretical tool for statistical classification.

• Naive Bayes Classifier: This is a simple, powerful algorithm used for tasks like spam filtering. The classifier is literally programmed to calculate $P(\text{Class} | \text{Features})$.
  $$P(\text{Spam} | \text{''Free Money''})$$
  It uses Bayes' Theorem to find the probability of the class (Spam) given the feature (word "Free Money"), relying on the historical probabilities (Priors) and the likelihood calculated from the training data.
• Model Evaluation: The probabilities calculated by the Law of Total Probability are the foundation for evaluating classifiers. The confusion matrix (Lesson 3.9) uses terms like Sensitivity and Specificity, which are all based on reversing conditional probabilities.

3.2 The Quant Finance Connection: Bayesian Statistics

While the core of this curriculum is Frequentist (Module 3/4), Bayesian statistics (which uses Bayes' Theorem as its core principle) is a powerful alternative, especially in finance.

• Incorporating Priors: A Bayesian analyst starts with a Prior Belief ($P(B)$) about a trading strategy's expected return. They then collect data ($A$) and use Bayes' Theorem to calculate the Posterior Probability ($P(B | A)$). This method is useful because it allows quants to incorporate human judgment or information from related markets (the Prior) into the final estimate, rather than relying solely on the sample data.
• Event Risk Modeling: Insurance and risk models use the Law of Total Probability to forecast low-probability events. For example, the total probability of a catastrophic loss ($A$) is calculated by summing the loss probability conditional on different, known fault modes ($B_i$: earthquake, power failure, cyberattack).