Image Segmentation: Techniques, Algorithms & Real-World Applications

TAKEAWAYS:

• Segmentation splits images into meaningful regions (objects/background) using pixel similarity, edges, or clustering—critical for object recognition in apps like Pathao’s traffic analysis or NTC’s license plate detection.
• Thresholding (global/local) and edge detection (Sobel, Canny) are fast but fail on uneven lighting; region-growing and clustering (K-means) handle complex scenes better.
• Morphological operations (erosion/dilation) clean up noise but distort shapes—use them after segmentation (e.g., Daraz’s product bounding boxes).
• Worked examples show how to compute thresholds, trace edge-detection steps, and compare segmentation accuracy using real images (e.g., a blurry traffic sign).
• Exam focus: Define methods, draw block diagrams (e.g., Canny edge detector pipeline), and calculate segmentation metrics (e.g., IoU for a 5×5 grid).
• Real-world tie: WhatsApp’s photo filters use segmentation to isolate faces; Nepal Police’s drone surveillance segments suspicious vehicles from backgrounds.

1. What Is Image Segmentation?

Segmentation divides an image into discrete, meaningful regions (e.g., foreground objects, textures). Unlike enhancement (which improves pixel values), segmentation groups pixels into logical units for analysis.

graph TD
    A["Input Image"] --> B["Preprocessing\n(Noise Reduction)"]
    B --> C["Segmentation\n(Pixel/Region/Edge-based)"]
    C --> D["Post-processing\n(Morphology, Merging)"]
    D --> E["Output:\nLabeled Regions"]

Why it matters:

• Object detection: Pathao’s app segments riders from traffic to optimize routes.
• Medical imaging: Doctors segment tumors in X-rays for surgery planning.
• Autonomous vehicles: Ncell’s 5G-enabled cars segment pedestrians from roads.

2. Key Segmentation Methods

A. Pixel-Based: Thresholding

Idea: Split pixels into foreground/background using a global or local threshold. How it works:

1. Choose a threshold $T$. Pixels $\ge T$ = object; $<T$ = background.
2. Global thresholding (e.g., Otsu’s method) picks $T$ for the whole image.
3. Local thresholding adapts $T$ per region (e.g., Bernsen’s method for uneven lighting).

Worked Example: Otsu’s Thresholding Given a histogram:

Steps:

1. Compute class probabilities $P_i=\text{pixels at level }i/\text{total pixels}$. $P_0=300/4000=0.075$, $P_1=0.125$, etc.
2. Calculate weighted mean ${\mu }_i={\sum }_{j=0}^{i}j\cdot P_j$. ${\mu }_0=0$, ${\mu }_1=0.125$, ${\mu }_2=0.375$, etc.
3. Find $T$ that maximizes inter-class variance: ${\sigma }^2(T)=w_0(T)\cdot w_1(T)\cdot ({\mu }_0(T)-{\mu }_1(T){)}^2$, where $w_0(T)={\sum }_{i=0}^T{P}_i$, $w_1(T)=1-w_0(T)$.

Result: $T=3$ (highest ${\sigma }^2$). Output:

graph TD
    A["Input Image"] -->|"T=3"| B["Segmented\nObject (gray ≥3)"]
    B --> C["Background\n(gray <3)"]

Limitations:

• Fails on multi-modal images (e.g., blood cells in microscopy).
• Real fix: Use adaptive thresholding (e.g., for NTC’s license plate scans with shadows).

B. Edge-Based: Detecting Boundaries

Idea: Find edges (sudden intensity changes) to outline objects. Operators:

Worked Example: Sobel Edge Detection For a 3×3 pixel patch:

[ 100 120 100 ]
[ 120 150 120 ]
[ 100 120 100 ]

Steps:

1. Apply Sobel kernels: $G_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]$, $G_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]$.
2. Compute gradients: $G_x=(100\cdot -1+120\cdot 0+100\cdot 1)+\cdots =40$, $G_y=(100\cdot -1+120\cdot -2+100\cdot 1)+\cdots =-60$.
3. Magnitude: $\sqrt{40^2+(-60{)}^2}=\sqrt{5200}\approx 72$.
4. Threshold: If $\ge 50$, mark as edge.

Canny Edge Detector Pipeline (5 steps):

flowchart LR
    A["1. Noise Reduction\n(Gaussian blur)"] --> B["2. Gradient Calc\n(Sobel)"]
    B --> C["3. Non-Max Suppression\n(Keep local maxima)"]
    C --> D["4. Double Thresholding\n(Hysteresis)"]
    D --> E["5. Edge Tracking\n(Connect regions)"]

Why Canny?

• Used in Google Maps’ street view segmentation to detect lanes.
• Exam tip: Draw this pipeline and label each step!

C. Region-Based: Grouping Pixels

Idea: Merge adjacent pixels with similar properties (intensity, texture). Methods:

Worked Example: Region Growing

[ 50 52 60 ]
[ 55 58 70 ]
[ 60 65 80 ]

Steps:

1. Pick seed at (1,1) = 58, threshold $T=5$.
2. Compare neighbors:
   • (1,2) = 52: $|58-52|=6>5$ → reject.
   • (2,1) = 55: $|58-55|=3\le 5$ → accept.
3. Repeat until no new pixels join.

Output:

[ X  X  X ]
[ A  A  X ]
[ A  A  X ]

($A$: segmented region, $X$: background)

Real Use: Daraz’s product images use region growing to isolate items from white backgrounds.

D. Clustering: K-Means Segmentation

Idea: Group pixels into $K$ clusters (e.g., $K=3$: sky, road, car). Steps:

1. Randomly pick $K$ centroids.
2. Assign each pixel to the nearest centroid.
3. Recompute centroids as the mean of assigned pixels.
4. Repeat until convergence.

Worked Example: Segment a 2×2 image:

[ (50,100) (150,50) ]
[ (200,200) (50,150) ]

For $K=2$:

1. Initial centroids: $C_1=(50,100)$, $C_2=(200,200)$.
2. Assign pixels:
   • (50,100) → $C_1$,
   • (150,50) → $C_2$ (closer to $C_2$),
   • (200,200) → $C_2$,
   • (50,150) → $C_1$.
3. Recompute centroids: $C_1=\left(\frac{50+50}{2},\frac{100+150}{2}\right)=(50,125)$, $C_2=\left(\frac{150+200}{2},\frac{50+200}{2}\right)=(175,125)$.
4. Repeat until centroids stabilize.

Exam Tip: Always show initialization → assignment → update steps!

3. Morphological Operations for Cleanup

After segmentation, use erosion/dilation to remove noise or fill gaps. Definitions:

• Erosion: Shrinks objects (removes small regions). $A\ominus B=\{z|B_z\subseteq A\}$.
• Dilation: Expands objects (fills holes). $A\oplus B=\{z|{\hat{B}}_z\cap A\ne \varnothing \}$.

Worked Example: Clean up a segmented license plate. Input:

[ 1 1 0 0 ]
[ 1 1 1 0 ]
[ 0 1 1 1 ]

Step 1: Erode with structuring element $B=\left[\begin{array}{c}1\\ 1\end{array}\right]$:

• Center pixel (2,2) = 1: $B$ fits → keep 1.
• Edge pixels (e.g., (1,1)): $B$ doesn’t fit → set to 0. Output:

[ 0 0 0 0 ]
[ 0 1 0 0 ]
[ 0 0 1 0 ]

Step 2: Dilate to recover shape:

• Use same $B$, output becomes:

[ 0 1 0 0 ]
[ 1 1 1 0 ]
[ 0 1 1 1 ]

Real Use: NTC’s automated number plate reader uses erosion to remove dust spots before OCR.

4. Evaluation Metrics

Quantify segmentation quality with:

Worked Example: Compare two segmentations of a 2×2 grid. Ground Truth:

[ 1 0 ]
[ 0 1 ]

Prediction A:

[ 1 1 ]
[ 0 1 ]

Prediction B:

[ 1 0 ]
[ 1 0 ]

Calculations:

## In the Real World

1. Pathao’s Ride-Hailing App

   • Idea Used: Edge detection (Canny) + region growing
   • How: Segments riders from traffic in real-time camera feeds to avoid collisions. Uses Gaussian blur → Canny edges → region merging to track moving objects.

2. Nepal Police’s Drone Surveillance

   • Idea Used: K-means clustering for color segmentation
   • How: Drones capture crowd images, then cluster pixels into $K=3$ groups (people, vehicles, background) to detect illegal gatherings. Exam tie-in: If asked about clustering, mention this!

3. WhatsApp’s Photo Filters

   • Idea Used: Skin tone segmentation (thresholding + morphological ops)
   • How: Filters like "Dog Face" use Otsu’s thresholding to isolate faces, then apply effects only to segmented regions. Real numbers: For a 100×100 face image, threshold $T=120$ (skin tone ~100–150 gray).

4. NTC’s License Plate Reader

   • Idea Used: Adaptive thresholding + erosion
   • How: Scans plates under varying lighting. Step 1: Bernsen’s thresholding handles shadows. Step 2: Erosion removes dust spots. Exam tip: Draw a block diagram of this pipeline!

## Exam Tip

1. Definitions: Always define methods with maths (e.g., $A\ominus B$ for erosion).
2. Diagrams: Draw block diagrams for:
   • Canny edge detector (5 steps).
   • Region-growing pipeline.
3. Calculations: Practice:
   • Otsu’s thresholding on histograms.
   • Sobel/Canny edge detection on 3×3 patches.
   • IoU/Dice scores for 2×2 grids.
4. Comparisons: Memorize this table for short-answer questions:

   Method	Speed	Accuracy	Best For
   Thresholding	Fast	Low	Uniform lighting
   Canny	Medium	High	Medical/automotive
   K-means	Slow	Medium	Color/texture scenes

5. Real-world links: If asked about applications, name Pathao, NTC, or WhatsApp and tie to the method (e.g., "Pathao uses Canny edges for collision avoidance").

## Past Exam Questions Solved

Q: "From the given gray-level histogram, enhance the image using histogram equalization." Solution:

1. Compute CDF (cumulative distribution function): $CDF(k)={\sum }_{i=0}^k{P}_i$. For $k=2$: $CDF(2)=0.075+0.125+0.25=0.45$.
2. Scale $CDF$ to 0–255: $s_k=CDF(k)\times 255$. For $k=2$: $s_2=0.45\times 255\approx 115$.
3. Map original gray levels to new values:
   • Original 0 → 0,
   • Original 1 → 64 (since $CDF(1)=0.2$),
   • Original 2 → 115, etc.

Q: "Explain region-based segmentation with an example." Answer:

Region-based segmentation groups pixels into homogeneous regions using criteria like intensity or texture. For example, in a 3×3 image with center pixel 50 and neighbors [48, 52, 55], starting from the center and using a threshold of 5, we merge all pixels within $\pm 5$ of 50. This is used in Daraz’s product segmentation to isolate items from backgrounds. Visual: Draw a 3×3 grid with merged regions shaded.

## Summary Checklist

Before the exam, ensure you can:

1. Draw the Canny edge detector pipeline.
2. Calculate Otsu’s threshold for a given histogram.
3. Compare thresholding vs. clustering methods in a table.
4. Apply erosion/dilation to a 3×3 binary image.
5. Name 2 real-world apps using segmentation (e.g., Pathao, NTC).
6. Compute IoU for a 2×2 segmentation example.