Bayes’ Theorem, Probabilistic Reasoning & Decision Trees

TAKEAWAYS:

• Bayes’ theorem updates probabilities when new evidence arrives, enabling AI to make smarter decisions under uncertainty.
• Conditional probability ($P(A|B)$) measures how likely event $A$ is given event $B$, forming the backbone of probabilistic reasoning.
• Decision trees model choices and outcomes, balancing risk and reward in AI-driven decisions (e.g., loan approvals, medical tests).
• Naïve Bayes simplifies calculations by assuming feature independence, widely used in spam detection and sentiment analysis.
• Expected utility theory helps AI agents choose actions that maximize long-term benefits, even with incomplete information.
• Real-world applications include fraud detection (Khalti), disease prediction (healthcare apps), and route optimization (Pathao).

---

1. Why Probability in AI?

AI systems often operate in uncertain environments—where data is noisy, incomplete, or probabilistic. For example:

• A spam filter cannot be 100% sure an email is spam.
• A doctor cannot diagnose a disease with absolute certainty.
• A self-driving car must make decisions despite sensor errors.

Probabilistic reasoning allows AI to handle uncertainty by assigning degrees of belief (probabilities) to possible outcomes.

Key Concepts

• Probability ($P(A)$): Likelihood of event $A$ occurring.
• Conditional Probability ($P(A|B)$): Probability of $A$ given $B$ has occurred.
• Joint Probability ($P(A,B)$): Probability of both $A$ and $B$ happening together.
• Independent Events: $P(A,B)=P(A)\cdot P(B)$ (e.g., rolling a die and flipping a coin).

---

2. Bayes’ Theorem: Updating Beliefs with Evidence

Bayes’ theorem formalizes how to update probabilities when new evidence arrives. It is the foundation of probabilistic reasoning in AI.

Formula

$$P(H|E)=\frac{P(E|H)\cdot P(H)}{P(E)}$$

• $P(H|E)$: Posterior probability (belief after seeing evidence $E$).
• $P(E|H)$: Likelihood (how likely $E$ is if $H$ is true).
• $P(H)$: Prior probability (initial belief before seeing $E$).
• $P(E)$: Marginal probability (total probability of $E$, often computed as $P(E|H)P(H)+P(E|\neg H)P(\neg H)$).

Worked Example: Medical Diagnosis

Suppose a disease affects 1% of the population ($P(\text{Disease})=0.01$). A test is 95% accurate:

• If a person has the disease, the test is positive 95% of the time ($P(\text{Positive}|\text{Disease})=0.95$).
• If a person does not have the disease, the test is negative 90% of the time ($P(\text{Negative}|\neg \text{Disease})=0.90$).

Question: If a randomly selected person tests positive, what is the probability they actually have the disease?

Solution

1. Define events:

   • $H$: Person has the disease.
   • $E$: Test is positive.

2. Given:

   • $P(H)=0.01$ (prior).
   • $P(E|H)=0.95$ (likelihood if diseased).
   • $P(E|\neg H)=1-0.90=0.10$ (false positive rate).

3. Compute $P(E)$ (marginal probability): $$P(E)=P(E|H)P(H)+P(E|\neg H)P(\neg H)=(0.95\times 0.01)+(0.10\times 0.99)=0.0095+0.099=0.1085$$

4. Apply Bayes’ theorem: $$P(H|E)=\frac{P(E|H)\cdot P(H)}{P(E)}=\frac{0.95\times 0.01}{0.1085}\approx 0.0876(\text{or }8.76\%)$$

Surprising Result: Even with a positive test, the probability of having the disease is only ~8.76%! This is because the disease is rare ($P(H)=1\%$), and false positives dominate.

---

3. Conditional Probability and Independence

Conditional Probability Table Example

Consider a spam detection system that checks two features:

• Feature 1 (F1): Email contains the word "FREE".
• Feature 2 (F2): Email has urgent language.

Question: What is $P(S|F1,F2)$ (probability email is spam given both features)?

Solution: $$P(S|F1,F2)=\frac{P(F1,F2|S)P(S)}{P(F1,F2)}=\frac{0.6\times 0.5}{0.6\times 0.5+0.02\times 0.5}=\frac{0.3}{0.3+0.01}\approx 0.968(\text{or }96.8\%)$$

Independence Check

Two events $A$ and $B$ are independent if: $$P(A,B)=P(A)\cdot P(B)$$ Example: Rolling a die and flipping a coin are independent. Non-example: Rain and carrying an umbrella are not independent (if it rains, you’re more likely to carry an umbrella).

---

4. Naïve Bayes Classifier: Simplifying Probabilistic Models

Naïve Bayes assumes features are conditionally independent given the class. This simplifies calculations while remaining effective.

Formula

For a classification problem with features $X=\{X_1,X_2,...,X_n\}$ and classes $C$: $$P(C|X_1,X_2,...,X_n)\propto P(C)\cdot \prod _{i=1}^{n}P(X_i|C)$$

Worked Example: Sentiment Analysis

A sentiment analysis model classifies tweets as Positive ($P$) or Negative ($N$) based on words:

• Words: "love", "hate".
• Data:
  • $P(\text{Positive})=0.6$, $P(\text{Negative})=0.4$.
  • $P(\text{"love"}|\text{Positive})=0.7$, $P(\text{"love"}|\text{Negative})=0.1$.
  • $P(\text{"hate"}|\text{Positive})=0.1$, $P(\text{"hate"}|\text{Negative})=0.8$.

Tweet: "I love this product but hate the delivery." Question: What is the probability this tweet is Positive?

Solution

1. Extract features: "love" (present), "hate" (present).
2. Compute likelihoods:
   • $P(\text{"love"},\text{"hate"}|\text{Positive})=P(\text{"love"}|\text{Positive})\cdot P(\text{"hate"}|\text{Positive})=0.7\times 0.1=0.07$.
   • $P(\text{"love"},\text{"hate"}|\text{Negative})=0.1\times 0.8=0.08$.
3. Apply Naïve Bayes: $$P(\text{Positive}|\text{"love","hate"})\propto P(\text{Positive})\cdot P(\text{"love"},\text{"hate"}|\text{Positive})=0.6\times 0.07=0.042$$ $$P(\text{Negative}|\text{"love","hate"})\propto P(\text{Negative})\cdot P(\text{"love"},\text{"hate"}|\text{Negative})=0.4\times 0.08=0.032$$
4. Normalize: $$P(\text{Positive})=\frac{0.042}{0.042+0.032}\approx 0.567(\text{or }56.7\%)$$

Conclusion: The tweet is more likely Negative (43.3%) than Positive (56.7%), despite containing "love". This shows how context matters even in Naïve Bayes!

---

5. Decision Trees for Risk-Based Decisions

Decision trees model sequential decisions under uncertainty, balancing risks and rewards.

Key Components

1. Root Node: Starting point (e.g., "Should I invest?").
2. Branches: Possible actions or outcomes.
3. Leaf Nodes: Final outcomes with associated probabilities.
4. Utility Values: Numerical values representing desirability (e.g., profit, cost).

Example: Loan Approval System (Like NMB Bank)

A bank uses a decision tree to approve loans based on:

• Credit Score (High/Low).
• Income Level (High/Low).

Data:

• $P(\text{High Credit})=0.6$, $P(\text{Low Credit})=0.4$.
• $P(\text{High Income})=0.5$, $P(\text{Low Income})=0.5$.
• Default rates:
  • High Credit + High Income: 5% default.
  • High Credit + Low Income: 15% default.
  • Low Credit + High Income: 30% default.
  • Low Credit + Low Income: 60% default.

Decision Tree:

Question: Should the bank approve a loan for a customer with High Credit + Low Income?

Solution:

• Expected Loss if approved: $$0.15\times \text{Loan Amount}=15\%\text{ chance of default.}$$
• Expected Profit if rejected: 0 (no business).
• Decision: Approve if expected profit > expected loss (e.g., if loan amount is small).

---

6. Expected Utility Theory

AI agents should maximize expected utility, not just probability. Utility combines:

• Probability of outcomes.
• Value (utility) of outcomes.

Formula

$$\text{Expected Utility}=\sum P({\text{Outcome}}_i)\times \text{Utility}({\text{Outcome}}_i)$$

Example: Investment Decision

An investor has two options:

1. Safe Investment: 100% chance of $10,000 profit.
2. Risky Investment: 50% chance of $50,000 profit, 50% chance of $0.

Question: Which should they choose if they are risk-neutral?

Solution:

• Safe: $1.0\times 10,000=10,000$.
• Risky: $0.5\times 50,000+0.5\times 0=25,000$. Decision: Choose the risky investment (higher expected utility).

---

In the Real World

1. Khalti Fraud Detection

   • Uses Bayes’ theorem to calculate $P(\text{Fraud}|\text{Transaction Features})$.
   • Example: If a transaction has high amount + unusual location, the system flags it as likely fraud ($P(\text{Fraud}|\text{High Amount, New Location})\gg P(\text{Fraud})$).

2. Pathao Driver Routing

   • Uses decision trees to balance:
     • Distance to pickup.
     • Traffic conditions.
     • Driver’s current location.
   • Example: If $P(\text{Traffic Jam})>0.7$, the system reroutes to avoid delays.

3. Nepal Electricity Load Prediction (NTC)

   • Uses Naïve Bayes to predict peak demand based on:
     • Temperature.
     • Time of day.
     • Historical usage.
   • Example: If $P(\text{High Demand}|\text{Summer, 6 PM})=0.9$, NTC prepares extra power.

---

7. Comparing Probabilistic Models

---

Exam Tip

1. Bayes’ Theorem Questions:

   • Always define events clearly ($H$, $E$).
   • Compute $P(E)$ using the law of total probability.
   • Common pitfall: Ignoring $P(E)$ and directly computing $P(H|E)$.

2. Naïve Bayes:

   • Assume independence to simplify calculations.
   • Use log probabilities to avoid underflow with small numbers.

3. Decision Trees:

   • Draw the tree explicitly for step-by-step marks.
   • Calculate expected values at each node.

4. Real-World Applications:

   • Link concepts to Nepali examples (e.g., Khalti fraud, Pathao routing).
   • Explain why probability is needed (uncertainty in real data).

---

8. Common Mistakes to Avoid

• Ignoring prior probabilities: Always use $P(H)$, not just $P(E|H)$.
• Assuming independence when none exists: Naïve Bayes fails if features are correlated.
• Misapplying Bayes’ theorem: $P(H|E)\ne P(E|H)$.
• Overcomplicating decision trees: Start with a simple binary tree before adding complexity.

---

9. Practice Problems

1. Medical Testing:

   • A test for a rare disease (prevalence = 2%) has 99% true positive rate and 95% true negative rate.
   • If a person tests positive, what is $P(\text{Disease}|\text{Positive})$?

2. Spam Filter:

   • Words "FREE" and "WIN" appear in 80% of spam and 5% of non-spam.
   • If an email contains both, what is $P(\text{Spam}|\text{FREE, WIN})$?

3. Investment Decision:

   • Option A: 70% chance of $10,000, 30% chance of $0.
   • Option B: 100% chance of $7,000.
   • Which maximizes expected utility?

---

10. Summary Visual

---

Final Note: Probabilistic reasoning is everywhere in AI—from fraud detection to healthcare. Master Bayes’ theorem, Naïve Bayes, and decision trees, and you’ll excel in both exams and real-world applications!