Rosenblatt’s Perceptron: Binary Classification, Learning Rules & Limitations

TAKEAWAYS:

• The Perceptron is a single-layer neural network that classifies linearly separable binary data using a weighted sum and step activation.
• It learns via the Perceptron Learning Rule (PLR), adjusting weights iteratively to minimize classification errors.
• Geometrically, the Perceptron finds the optimal hyperplane that separates two classes in the feature space.
• It fails on non-linearly separable problems (e.g., XOR), motivating later architectures like MLPs.
• Real-world use: Spam detection (e.g., eSewa transaction alerts), simple fraud detection (Khalti), and binary decision systems (NTC network routing).
• Limitations: No hidden layers, restricted to linear boundaries, and requires fully separable training data.

1. What is a Perceptron?

A Perceptron is the simplest feedforward neural network for binary classification. It mimics a biological neuron:

• Takes input features $x_1,x_2,...,x_n$ (e.g., pixel intensities, transaction amounts).
• Computes a weighted sum $z=w_1{x}_1+w_2{x}_2+...+w_n{x}_n+b$ (where $b$ is the bias).
• Applies a step (threshold) activation function: $$y=\{\begin{array}{ll}1& \text{if }z\ge 0\\ 0& \text{otherwise}\end{array}$$
• Outputs 1 (class A) or 0 (class B).

Visual: Perceptron Structure

2. Mathematical Formulation

Input-Output Relationship

For inputs $\mathbf{x}=[x_1,x_2,...,x_n{]}^T$ and weights $\mathbf{w}=[w_1,w_2,...,w_n{]}^T$: $$z={\mathbf{w}}^T\mathbf{x}+b$$ y = \begin{cases} 1 & \text{if } \mathbf{w}^T \mathbf{x} + b \geq 0 \\ 0 & \text{otherwise} \end \end{cases} Key Idea: The Perceptron finds a decision boundary (hyperplane) ${\mathbf{w}}^T\mathbf{x}+b=0$ that separates the two classes.

Geometric Interpretation

In 2D, the decision boundary is a line: $$w_1{x}_1+w_2{x}_2+b=0$$ In 3D, it’s a plane, and in $n$-dimensions, a hyperplane.

3. Perceptron Learning Rule (PLR)

The Perceptron learns by adjusting weights to minimize classification errors. The update rule: $${\mathbf{w}}_{new}={\mathbf{w}}_{old}+\eta (\mathbf{x}-y{\mathbf{w}}_{old})$$ where:

• $\eta$ = learning rate (typically $0<\eta \le 1$),
• $y$ = true label (0 or 1),
• $\hat{y}$ = predicted output.

Steps:

1. Compute $z={\mathbf{w}}^T\mathbf{x}+b$.
2. Predict $\hat{y}=\text{sign}(z)$.
3. If $\hat{y}\ne y$ (misclassification), update weights: $$\mathbf{w}=\mathbf{w}+\eta (y-\hat{y})\mathbf{x}$$ $$b=b+\eta (y-\hat{y})$$
4. Repeat for all training examples (epochs).

Worked Example: AND Gate as a Perceptron

Train a Perceptron to implement the AND logic gate:

Initial weights: $\mathbf{w}=[0,0{]}^T$, $b=0$, $\eta =0.1$.

Epoch 1:

• Input (1,1), $y=1$: $z=0+0+0=0$ → $\hat{y}=0$ (error). Update: $$\mathbf{w}=[0,0]+0.1(1-0)[1,1]=[0.1,0.1]$$ $b=0+0.1(1-0)=0.1$.

• Input (1,0), $y=0$: $z=0.1(1)+0.1(0)+0.1=0.2$ → $\hat{y}=1$ (error). Update: $$\mathbf{w}=[0.1,0.1]+0.1(0-1)[1,0]=[0,0.1]$$ $b=0.1+0.1(0-1)=0$.

• Input (0,1), $y=0$: $z=0(0)+0.1(1)+0=0.1$ → $\hat{y}=1$ (error). Update: $$\mathbf{w}=[0,0.1]+0.1(0-1)[0,1]=[0,0]$$ $b=0+0.1(0-1)=-0.1$.

Epoch 2: Continue until all inputs are classified correctly. Final weights might converge to $\mathbf{w}=[1,1]$, $b=-1.5$ (varies by $\eta$).

Visual: Convergence Path

flowchart TD
    A["Epoch 1: w=[0.1,0.1], b=0.1"] --> B["Epoch 2: w=[0.5,0.5], b=-0.1"]
    B --> C["Epoch 3: w=[0.9,0.9], b=-1.0"]
    C --> D["Converged: w≈[1,1], b=-1.5"]

4. Real-World Applications

1. eSewa Transaction Fraud Detection

• Problem: Classify transactions as fraudulent (1) or legitimate (0).
• How Perceptron Helps:
  • Inputs: Transaction amount, time, location, user history.
  • Trained on historical data to flag suspicious patterns (e.g., high amount + unusual time).
  • Limitation: Fails if fraudsters use non-linear patterns (e.g., encoded amounts).

2. Khalti Payment Routing

• Problem: Route payments to the correct bank or wallet (binary: success/failure).
• How Perceptron Helps:
  • Inputs: Account number format, balance, transaction type.
  • Learns to reject invalid routes (e.g., insufficient balance + wrong format).
  • Real Example: Khalti’s early fraud detection used Perceptrons before switching to deeper networks.

3. NTC Network Packet Classification

• Problem: Classify network packets as legitimate (0) or malicious (1).
• How Perceptron Helps:
  • Inputs: Packet size, source IP, port number.
  • Acts as a simple firewall rule (e.g., block packets from known malicious IPs).
  • Limitation: Cannot detect encrypted malware (requires MLPs or transformers).

5. Limitations of the Perceptron

The XOR Problem: Why Perceptrons Fail

The XOR truth table is not linearly separable:

Visual: XOR Data in 2D

Solution: Use a Multilayer Perceptron (MLP) (covered in Unit 5) with hidden layers to model non-linear boundaries.

6. Comparison: Perceptron vs. Other Models

7. Exam Tip: How to Score Full Marks

1. Define Clearly:

   • Start with: "A Perceptron is a single-layer neural network for binary classification using a weighted sum and step activation."
   • Mention inputs, weights, bias, and output in your answer.

2. Show the Math:

   • Write the weight update rule and decision boundary equation.
   • For worked examples, show all steps (e.g., AND gate training).

3. Draw the Geometry:

   • Sketch a 2D scatter plot with the decision boundary line.
   • Label axes as features and classes.

4. Discuss Limitations:

   • Mention XOR problem and linear separability.
   • Compare with MLPs or kernel methods (Unit 6).

5. Real-World Tie-In:

   • Link to eSewa/Khalti fraud detection or NTC routing.
   • Example: "A Perceptron in eSewa could flag transactions where amount > threshold AND time = night."

6. Common Pitfalls:

   • ❌ Forgetting to normalize inputs (affects convergence).
   • ❌ Assuming Perceptrons work for all problems (they don’t!).
   • ❌ Confusing PLR with gradient descent (they’re different).

Final Note: The Perceptron is the building block of neural networks. Master its math, geometry, and limitations to ace Unit 2—and prepare for Units 5 (MLPs) and 6 (kernels) where these ideas evolve!