Bayesian Learning: Probability, Naive Bayes, MAP, EM, and Applications

Key Concepts in Bayesian Learning

1. Bayesian Probability and Conditional Probability

Bayesian learning is rooted in conditional probability, which quantifies how the probability of an event changes given new evidence. The core formula is Bayes’ Theorem:

$$P(A|B)=\frac{P(B|A)\cdot P(A)}{P(B)}$$

• $P(A|B)$: Posterior probability (what we want to find).
• $P(B|A)$: Likelihood (how likely the evidence is given the hypothesis).
• $P(A)$: Prior probability (our initial belief about the hypothesis).
• $P(B)$: Marginal probability (total probability of the evidence).

Example: Spam Detection in eSewa

Suppose eSewa wants to classify messages as spam (S) or not spam (¬S). We use Bayes’ Theorem to compute the probability that a message is spam given it contains the word "discount" ($D$):

$$P(S|D)=\frac{P(D|S)\cdot P(S)}{P(D)}$$

Given:

• $P(S)=0.3$ (30% of messages are spam).
• $P(D|S)=0.5$ (50% of spam messages contain "discount").
• $P(D|\neg S)=0.05$ (5% of non-spam messages contain "discount").

Compute $P(D)$ (marginal probability): $$P(D)=P(D|S)\cdot P(S)+P(D|\neg S)\cdot P(\neg S)=0.5\times 0.3+0.05\times 0.7=0.15+0.035=0.185$$

Final posterior probability: P(S|D) = \frac{0.5 \times 0.3}{0.185} \approx 0.81 \quad (\text{81% chance of spam})

2. Naive Bayes Classifier

Naive Bayes assumes feature independence (e.g., words in a document are independent given the class). It is widely used for text classification, spam filtering, and sentiment analysis.

Types of Naive Bayes:

Example: Medical Diagnosis (Nepal Health App)

Suppose a health app predicts malaria (M) based on symptoms: fever (F), headache (H), and chills (C).

Given:

• $P(M)=0.1$ (10% of patients have malaria).
• $P(F|M)=0.9$, $P(H|M)=0.8$, $P(C|M)=0.7$.
• $P(F|\neg M)=0.3$, $P(H|\neg M)=0.2$, $P(C|\neg M)=0.1$.

Compute $P(M|F,H,C)$: $$P(M|F,H,C)=\frac{P(F,H,C|M)\cdot P(M)}{P(F,H,C)}$$ Assuming independence: $$P(F,H,C|M)=P(F|M)\cdot P(H|M)\cdot P(C|M)=0.9\times 0.8\times 0.7=0.504$$ $$P(F,H,C|\neg M)=0.3\times 0.2\times 0.1=0.006$$ $$P(F,H,C)=P(F,H,C|M)\cdot P(M)+P(F,H,C|\neg M)\cdot P(\neg M)=0.504\times 0.1+0.006\times 0.9=0.0504+0.0054=0.0558$$ P(M|F, H, C) = \frac{0.504 \times 0.1}{0.0558} \approx 0.903 \quad (\text{90.3% chance of malaria})

3. Maximum A Posteriori (MAP) Estimation

MAP estimates the most probable parameter given data, balancing likelihood and prior:

$${\theta }_{MAP}=\arg \underset{\theta }{\max }P(\theta |X)=\arg \underset{\theta }{\max }P(X|\theta )\cdot P(\theta )$$

Example: Loan Approval (Nepal Bank)

A bank uses MAP to decide whether to approve a loan ($L$) based on income (I) and credit score (S).

Given:

• Prior: $P(L=\text{Yes})=0.6$, $P(L=\text{No})=0.4$.
• Likelihoods (from historical data):
  • $P(I=\text{High}|L=\text{Yes})=0.8$, $P(I=\text{High}|L=\text{No})=0.2$.
  • $P(S=\text{Good}|L=\text{Yes})=0.7$, $P(S=\text{Good}|L=\text{No})=0.3$.

Compute $P(L=\text{Yes}|I=\text{High},S=\text{Good})$: $$P(L=\text{Yes}|I,S)\propto P(I,S|L=\text{Yes})\cdot P(L=\text{Yes})$$ Assuming independence: $$P(I,S|L=\text{Yes})=P(I|L=\text{Yes})\cdot P(S|L=\text{Yes})=0.8\times 0.7=0.56$$ $$P(L=\text{Yes}|I,S)=\frac{0.56\times 0.6}{P(I,S)}(\text{Normalize later})$$ Since $0.56\times 0.6>0.2\times 0.3\times 0.4$, MAP approves the loan.

4. Expectation-Maximization (EM) Algorithm

EM is used for missing data problems (e.g., clustering with incomplete data). It alternates between:

1. E-step: Compute expected values of missing data.
2. M-step: Maximize the likelihood given current estimates.

Example: Customer Segmentation (Daraz)

Daraz wants to cluster customers into high-value (H) and low-value (L) groups, but some purchase data is missing.

Step 1 (E-step): Assume initial probabilities $P(H)=0.4$, $P(L)=0.6$. Compute expected responsibility of each customer for each cluster.

Step 2 (M-step): Update cluster means (e.g., average purchase amount) based on E-step results.

Repeat until convergence.

In the Real World

1. eSewa (Spam Filtering)

   • Uses Naive Bayes to classify transactions as fraudulent or legitimate by analyzing keywords like "urgent," "discount," and sender history.
   • Example: A message with "immediate payment" triggers a high spam score due to $P(\text{Spam}|\text{"immediate"})\approx 0.9$.

2. Nepal Rastra Bank (Loan Risk Assessment)

   • Applies MAP estimation to approve loans by balancing applicant credit history (prior) with current financials (likelihood).
   • Example: A farmer with a high credit score but low income may still get a loan if $P(\text{Approved}|\text{High Score, Low Income})$ is maximized.

3. Pathao (Ride Demand Prediction)

   • Uses Bayesian networks to predict rider demand in Kathmandu by combining historical data (prior) with real-time traffic (likelihood).
   • Example: If $P(\text{High Demand}|\text{Rain, Evening})$ is high, Pathao increases driver incentives.

Visualizing Bayesian Learning

1. Bayesian Network for Medical Diagnosis

graph TD
    A["Malaria (M)"] --> B["Fever (F)"]
    A --> C["Headache (H)"]
    A --> D["Chills (C)"]
    B --> E["Diagnosis"]
    C --> E
    D --> E

2. Naive Bayes Feature Independence

graph TD
    A["Class: Spam"] --> B["Word: Discount"]
    A --> C["Word: Urgent"]
    A --> D["Word: Free"]

3. EM Algorithm Convergence (Missing Data)

graph TD
    A["Initial Guess"] --> B["E-step: Compute Expectations"]
    B --> C["M-step: Update Parameters"]
    C --> D["Check Convergence?"]
    D -->|"No"| B
    D -->|"Yes"| E["Final Clusters"]

Advantages and Limitations

Exam Tip

1. Bayes’ Theorem: Always show all steps (prior, likelihood, marginal). Partial credit is given for correct setup.
2. Naive Bayes: Assume independence unless stated otherwise. Compare with logistic regression in exams.
3. MAP vs MLE: MAP includes a prior; MLE does not. State which is used when.
4. EM Algorithm: Explain E-step and M-step clearly. Use a small example (e.g., 2 clusters, 3 data points).
5. Real-world mapping: Relate problems to eSewa (spam), banks (loans), or health apps (diagnosis).

Final Worked Example: Traffic Light Optimization (NTC)

NTC wants to predict traffic jam (J) at a junction based on:

• Time (T): Peak (P) or Off-peak (O).
• Weather (W): Rain (R) or Sunny (S).

Given:

• $P(J)=0.4$, $P(\neg J)=0.6$.
• $P(P|J)=0.7$, $P(P|\neg J)=0.3$.
• $P(R|J)=0.6$, $P(R|\neg J)=0.1$.

Question: What is $P(J|P,R)$?

Solution: $$P(J|P,R)=\frac{P(P,R|J)\cdot P(J)}{P(P,R)}$$ Assuming independence: $$P(P,R|J)=P(P|J)\cdot P(R|J)=0.7\times 0.6=0.42$$ $$P(P,R)=P(P,R|J)\cdot P(J)+P(P,R|\neg J)\cdot P(\neg J)=0.42\times 0.4+(0.3\times 0.1)\times 0.6=0.168+0.018=0.186$$ P(J|P, R) = \frac{0.42 \times 0.4}{0.186} \approx 0.909 \quad (\text{90.9% chance of jam}) Action: NTC may extend the green light duration at this junction.