Neural Networks: Architectures, Learning & Applications

TAKEAWAYS:

• Neural networks are computational models inspired by biological neurons, organized into layers (input, hidden, output) that transform data through weighted connections.
• Forward propagation computes outputs, while backpropagation adjusts weights using gradient descent to minimize loss (e.g., mean squared error or cross-entropy).
• Key architectures include feedforward networks (static data), convolutional neural networks (CNNs) (images), and recurrent neural networks (RNNs/LSTMs) (sequences like text or time-series).
• Activation functions (ReLU, sigmoid, tanh) introduce non-linearity; loss functions measure prediction error to guide learning.
• Real-world applications span Nepali apps (e.g., Khalti’s fraud detection via anomaly detection networks) to global tech (Google’s image recognition, Ncell’s predictive maintenance).
• Challenges include vanishing gradients, overfitting, and computational cost, addressed via techniques like dropout, batch normalization, and hardware acceleration (GPUs/TPUs).

1. Biological Inspiration and Artificial Neurons

Neural networks emulate the human brain’s neurons, which communicate via electrical/chemical signals. An artificial neuron (perceptron) is the building block:

• Inputs: Received signals $x_1,x_2,...,x_n$ multiplied by weights $w_1,w_2,...,w_n$.
• Bias: A trainable offset $b$ (like a neuron’s resting potential).
• Activation: A non-linear function (e.g., $f(z)=\text{ReLU}(z)=\max (0,z)$) determines output.

graph LR
    A["Inputs \(x_1, x_2\)"] -->|"Weights w_1, w_2"| B["Summation: \(z = w_1x_1 + w_2x_2 + b\)"]
    B --> C["Activation: \(f(z)\)"]
    C --> D["Output"]

Why non-linearity? Linear functions (e.g., $z=w_1{x}_1+w_2{x}_2$) cannot model complex patterns. Activation functions enable hierarchical feature learning.

2. Neural Network Architectures

A. Feedforward Neural Networks (FNNs)

• Structure: Layers connected unidirectionally (input → hidden → output).
• Use case: Tabular data (e.g., predicting house prices from features like size, location).
• Example: A 3-layer FNN for classifying handwritten digits (MNIST dataset).

Worked Example: Predicting Daraz Order Delays

• Input: Order ID, distance (km), weather (rainy: 1, sunny: 0), time of day (hour).
• Hidden Layer: 8 neurons with ReLU activation.
• Output: Predicted delay (hours) via linear activation.
• Training Data:

  Order ID	Distance (km)	Rainy	Hour	Delay (hours)
  1001	5	1	18	2.1
  1002	3	0	10	0.5

Forward Pass Calculation: For Order 1001:

1. Input vector: $[5,1,18,1]$.
2. Hidden layer weights $W_1$ (random init), bias $b_1=0$. $z_1=W_1\cdot [5,1,18,1]+b_1$. Suppose $W_1=[[0.2,-0.3,0.1,0.4]]$ (1×4), then: $z_1=0.2\ast 5+(-0.3)\ast 1+0.1\ast 18+0.4\ast 1=1.0+(-0.3)+1.8+0.4=2.9$.
3. Apply ReLU: $a_1=\max (0,2.9)=2.9$.
4. Repeat for all 8 hidden neurons → output vector $[a_1,a_2,...,a_8]$.
5. Output layer weights $W_2$ (8×1) compute delay: $\text{delay}=W_2\cdot [a_1,...,a_8]+b_2$.

B. Convolutional Neural Networks (CNNs)

• Key Idea: Exploit local connectivity and parameter sharing for spatial data (images).
• Components:
  • Convolutional Layer: Applies filters (kernels) to detect features (edges, textures).
  • Pooling Layer: Reduces spatial dimensions (e.g., max pooling).
  • Fully Connected Layer: Classifies features (e.g., "cat" vs. "dog").

Real-World Example: Khalti’s Receipt Scanner

• Problem: Extract text from handwritten receipts.
• Solution: CNN to detect characters → OCR (Optical Character Recognition) to convert to digital text.
• Why CNN?
  • Filters learn to detect strokes (e.g., "₹" symbol) regardless of position/size.
  • Pooling reduces noise (e.g., smudged ink).

C. Recurrent Neural Networks (RNNs)

• Key Idea: Maintain hidden state to model sequences (time-series or text).
• Types:
  • Vanilla RNN: Suffer from vanishing gradients.
  • LSTM/GRU: Use gating mechanisms to retain long-term memory.

Worked Example: Ncell’s Charging Station Demand Prediction

• Input: Hourly data for 7 days (temperature, holidays, past demand).
• RNN Task: Predict demand at $t+1$ hour.
• LSTM Advantage: Captures patterns like "demand spikes on Fridays at 6 PM."

3. Activation Functions

Activation functions introduce non-linearity. Common choices:

Graph of Activation Functions: Why ReLU Dominates:

• Computationally efficient (no exponentials).
• Mitigates vanishing gradient problem (vs. sigmoid/tanh).

4. Loss Functions and Optimization

A. Loss Functions

Measure error between predictions ($\hat{y}$) and true values ($y$):

Example: NEPSE Stock Prediction

• Task: Predict tomorrow’s stock price (regression).
• Loss: MSE between predicted and actual price.
• Goal: Minimize $\text{MSE}=(500-490{)}^2+(480-475{)}^2=100+25=125$.

B. Optimization: Gradient Descent

Adjust weights to minimize loss using calculus:

1. Compute gradient of loss w.r.t. weights ($\frac{\partial \text{Loss}}{\partial W}$).
2. Update weights: $W_{\text{new}}=W_{\text{old}}-\alpha \cdot \nabla \text{Loss}$, where $\alpha$ = learning rate.

Variants:

• Batch Gradient Descent: Use entire dataset (slow but stable).
• Stochastic GD: Update per sample (noisy but fast).
• Mini-Batch GD: Compromise (e.g., batches of 32 samples).

Worked Example: Training a Simple Network

• Network: 1 input, 1 hidden neuron (ReLU), 1 output (linear).
• Data: $x=[1,2]$, $y=[2,4]$ (linear relationship $y=2x$).
• Initial Weights: $W_1=0.5$, $b_1=0$, $W_2=0.1$, $b_2=0$.
• Forward Pass for $x=1$:
  • Hidden: $z_1=0.5\ast 1+0=0.5$, $a_1=\text{ReLU}(0.5)=0.5$.
  • Output: $\hat{y}=0.1\ast 0.5+0=0.05$.
  • Loss (MSE): $(2-0.05{)}^2=3.9025$.
• Backpropagation:
  • $\frac{\partial \text{Loss}}{\partial W_2}=(0.05-2)\ast a_1=-1.95\ast 0.5=-0.975$.
  • Update $W_2$: $0.1-0.01\ast (-0.975)=0.10975$.

5. Training Neural Networks

A. Backpropagation Algorithm

1. Forward Pass: Compute predictions.
2. Compute Loss: Compare predictions to true values.
3. Backward Pass: Propagate error backward to compute gradients.
4. Update Weights: Adjust weights using gradients and learning rate.

Mermaid Diagram:

B. Challenges and Solutions

Example: Pathao’s Traffic Route Optimization

• Problem: Overfitting to training data (e.g., memorizing exact routes in Kathmandu).
• Solution: Dropout (randomly deactivate 20% of neurons during training) to generalize better.

6. Real-World Applications in Nepal and Globally

A. Nepali Examples

1. Khalti’s Fraud Detection

   • Idea: Anomaly detection using autoencoders (a type of neural network).
   • How: Train on normal transactions; flag transactions with high reconstruction error as fraudulent.
   • Output: Alerts like "Transaction ID 12345: 95% fraud probability."

2. Ncell’s Predictive Maintenance

   • Idea: LSTM networks analyze tower temperature/vibration data.
   • How: Predict equipment failure 24 hours in advance.
   • Impact: Reduces downtime by 30%.

3. eSewa’s Chatbot

   • Idea: RNN (or Transformer) for natural language understanding.
   • How: Processes queries like "Bill payment for NTC Rs. 500" and routes to the correct API.

B. Global Examples

1. Google’s Image Search (CNN)

   • Idea: Inception-v3 CNN processes 1000+ classes.
   • How: Detects objects in images (e.g., "cat" in a photo of a Siamese cat).

2. YouTube’s Recommendation System (Deep Learning)

   • Idea: Multi-layer perceptron (MLP) + collaborative filtering.
   • How: Predicts "You might also like" videos based on watch history.

3. Tesla’s Autopilot (CNN + RNN)

   • Idea: Combines spatial (CNN for camera input) and temporal (RNN for trajectory) data.
   • How: Detects pedestrians and predicts their movement.

7. Exam Tip

What to Expect in PU Exams:

1. Theory Questions (30%):

   • Define perceptron, backpropagation, and vanishing gradient problem.
   • Compare CNN vs. RNN architectures (use a table like above).
   • Explain why ReLU is preferred over sigmoid in hidden layers.

2. Numerical Problems (40%):

   • Forward/backward pass: Given weights and inputs, compute outputs or gradients. Example: "For a network with $W_1=[0.1,0.2]$, $b_1=0.3$, and input $x=[1,-1]$, compute the hidden layer output using ReLU."
   • Loss calculation: Compute MSE or cross-entropy for given predictions.
   • Weight updates: Perform one step of gradient descent.

3. Application-Based (30%):

   • Case Study: "How would you design a neural network for NTC’s customer churn prediction?" Structure:
     • Input: Customer data (call duration, complaints, usage).
     • Hidden: 2 layers (128 → 64 neurons, ReLU).
     • Output: 1 neuron (sigmoid) for "churn (1) or not (0)".
     • Loss: Binary cross-entropy.
   • Diagram: Draw a CNN for a given task (e.g., "design a network to classify Nepali handwritten digits").

Common Pitfalls:

• Forgetting to apply activation functions (e.g., using linear output for classification).
• Misaligning dimensions in matrix multiplications (e.g., input shape vs. weights).
• Confusing batch size (number of samples per update) with epochs (full passes over data).

Pro Tip:

• Memorize the shapes:
  • Input layer size = number of features.
  • Hidden layer size = hyperparameter (tune via validation error).
  • Output layer size = number of classes (for classification) or 1 (for regression).
• Practice: Use tools like TensorFlow Playground to visualize how layers/learning rates affect training.