Neural Networks: Basics, Models & Biological Inspiration

Biological Inspiration: The Brain vs. Artificial Neurons

The human brain contains ~86 billion neurons, each connected via ~100 trillion synapses. These biological neurons communicate via electrochemical signals (action potentials) and exhibit plasticity (adapting to new information). Artificial neural networks (ANNs) mimic this structure but simplify it:

Key Differences:

---

Artificial Neuron: The McCulloch-Pitts Model (1943)

The simplest ANN unit is the McCulloch-Pitts neuron, a binary classifier:

1. Inputs: $x_1,x_2,...,x_n$ (e.g., pixel intensities in an image).
2. Weights: $w_1,w_2,...,w_n$ (learned during training).
3. Bias: $b$ (threshold adjustment).
4. Activation: Step function $\varphi (z)=\{\begin{array}{ll}1& \text{if }z\ge 0\\ 0& \text{otherwise}\end{array}$, where $z={\sum }_{i=1}^n{w}_i{x}_i+b$.

Worked Example: Handwritten Digit Recognition Suppose a neuron classifies whether a pixel is part of a "1" (output=1) or "0" (output=0). For a 3-pixel input:

• Inputs: $x=[0.8,0.3,0.1]$ (gray-scale values).
• Weights: $w=[0.5,-0.2,0.4]$, bias $b=-0.1$.
• Calculation: $$z=(0.8\times 0.5)+(0.3\times -0.2)+(0.1\times 0.4)-0.1=0.4-0.06+0.04-0.1=0.28$$ Since $z=0.28\ge 0$, output $\varphi (z)=1$ ("1" detected).

---

Activation Functions: Nonlinearity in ANNs

Linear neurons (no activation) cannot model complex patterns. Activation functions introduce nonlinearity:

• Step Function: Binary threshold (used in McCulloch-Pitts).
• Sigmoid: $\sigma (z)=\frac{1}{1+e^{-z}}$ (outputs between 0 and 1).
• ReLU: $\text{ReLU}(z)=\max (0,z)$ (common in deep learning).
• Tanh: $\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$ (outputs between -1 and 1).

Graph Comparison:

Why ReLU Dominates:

• Computationally efficient (no exponentiation).
• Mitigates vanishing gradients in deep networks.
• Sparse activation (many neurons output 0).

---

Neural Network Architecture: Layers and Connectivity

ANNs are composed of layers:

1. Input Layer: Raw features (e.g., pixel values).
2. Hidden Layers: Intermediate computations (can be multiple).
3. Output Layer: Final prediction (e.g., class probabilities).

Fully Connected (Dense) Network:

Sparse Connectivity (e.g., CNNs):

• Neurons connect only to local regions (e.g., adjacent pixels).
• Reduces parameters and computational cost.

---

Training a Neural Network: Supervised Learning Basics

ANNs learn via gradient descent and backpropagation:

1. Forward Pass: Compute output for given inputs.
2. Loss Calculation: Compare output to true label (e.g., Mean Squared Error for regression).
3. Backward Pass: Adjust weights using the chain rule to minimize loss.

Loss Functions:

• MSE: $L=\frac{1}{n}{\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2$ (regression).
• Cross-Entropy: $L=-{\sum }_i{y}_i\log ({\hat{y}}_i)$ (classification).

Worked Example: Predicting House Prices Suppose a single neuron predicts price $\hat{y}$ from size $x$:

• True price $y=500,000$, predicted $\hat{y}=450,000$.
• MSE loss: $$L=(500,000-450,000{)}^2=25,000,000$$
• Gradient for weight $w$: $\frac{\partial L}{\partial w}=2(\hat{y}-y)\cdot x$.

---

In the Real World

1. eSewa (Nepal):

   • Idea Used: Multilayer Perceptrons (MLPs) for fraud detection.
   • How: ANNs analyze transaction patterns (time, amount, location) to flag suspicious activities (e.g., sudden large transfers). A hidden layer with ReLU activation learns non-linear relationships between features like "user’s usual spending" and "transaction amount."

2. Pathao (Ride-Hailing):

   • Idea Used: Dynamic Routing via Neural Networks.
   • How: A deep neural network predicts optimal driver routes in real-time by processing inputs like traffic data (from GPS), driver availability, and historical demand. The network’s output is a probability distribution over possible routes, and the highest-probability path is chosen. For example, during Kathmandu’s evening rush, the ANN might reroute drivers away from Thapathali via a hidden layer that detects congestion patterns.

3. Ncell’s Customer Churn Prediction:

   • Idea Used: Binary Classification with Sigmoid Output.
   • How: Ncell uses a neural network to predict whether a subscriber will cancel service. Inputs include call duration, data usage, and customer service interactions. The output layer has a single neuron with a sigmoid activation, outputting a probability (e.g., 0.87) that the customer will churn. If this exceeds a threshold (e.g., 0.8), the system triggers a retention offer.

---

Exam Tip

1. Define Clearly: Start with definitions (e.g., "An artificial neuron is a mathematical model inspired by biological neurons..."). Examiners check if you distinguish between biological neurons and ANNs.
2. Draw Diagrams: Sketch a 3-layer ANN (input, hidden, output) and label weights/biases. For partial credit, even a rough diagram with arrows is better than none.
3. Worked Examples: Always show step-by-step calculations for neuron outputs or loss functions. For example:
   • Given inputs/weights, compute $z$ and activation.
   • For MSE, write the formula and plug in numbers.
4. Compare Activation Functions: In short-answer questions, list one advantage and one disadvantage of ReLU vs. Sigmoid (e.g., ReLU is faster but can cause "dying ReLU" if weights are too large).
5. Real-World Links: If asked about applications, mention eSewa (fraud detection), Pathao (routing), or Ncell (churn prediction). Avoid vague answers like "ANNs are used in AI."
6. Common Pitfalls:
   • Forgetting the bias term in neuron calculations.
   • Confusing forward pass (prediction) with backward pass (learning).
   • Using the wrong activation for the task (e.g., ReLU for output in classification).

---

Summary Table: Key Concepts