Image Segmentation: Boundaries, Regions & Visual Perception

TAKEAWAYS:

• Image segmentation divides an image into homogeneous regions or boundaries (edges/lines) to simplify analysis.
• Boundary-based methods detect discontinuities (edges, corners) using gradient operators (Prewitt, Sobel, Canny).
• Region-based methods group pixels by similarity (intensity, texture, color) via splitting/merging or thresholding.
• Visual perception elements (Gestalt principles) guide human-like segmentation (proximity, similarity, closure).
• Distance metrics (Euclidean, Manhattan) measure pixel separations for region connectivity.
• Applications span medical imaging (tumor detection), autonomous driving (lane segmentation), and OCR (text extraction).

1. What is Image Segmentation?

• Object recognition (identifying cars in traffic).
• Medical diagnosis (segmenting tumors in MRI scans).
• Autonomous systems (detecting pedestrians for self-driving cars).

Why Segment Images?

• Simplification: Reduces complexity for higher-level tasks (e.g., counting objects).
• Feature extraction: Isolates regions for analysis (e.g., extracting blood vessels in retinal scans).
• Human-like perception: Mimics how humans group visual elements (e.g., seeing a "face" in a crowd).

2. Boundary-Based Segmentation: Detecting Discontinuities

Boundary-based methods identify edges, lines, or corners where pixel properties change abruptly. These discontinuities are classified into three types:

Types of Grey-Level Discontinuities

How to Detect Boundaries?

Boundary detection relies on gradient operators, which compute the rate of change in pixel intensity. Common operators:

1. Prewitt Operator: Uses 3×3 kernels to approximate gradients in horizontal and vertical directions.
2. Sobel Operator: Similar to Prewitt but with weighted kernels for smoother edges.
3. Canny Edge Detector: Multi-stage algorithm (noise reduction → gradient → non-maximum suppression → hysteresis thresholding).

Worked Example: Prewitt Edge Detection

Given Image (3×3 pixels):

| 0  | 30 | 60 |
| 5  | 32 | 62 |
| 10 | 38 | 64 |

Prewitt Kernels:

• Horizontal (Gx): [-1, 0, +1; -1, 0, +1; -1, 0, +1]
• Vertical (Gy): [-1, -1, -1; 0, 0, 0; +1, +1, +1]

Steps:

1. Compute Gx (horizontal gradient):

   (30+60+62) - (0+5+10) = 182 - 15 = 167
   (32+62+64) - (30+5+10) = 158 - 45 = 113
   (38+64) - (32+62) = 102 - 94 = 8  (partial, ignore)
   

   Gx ≈ [167, 113, 8] (simplified for center pixel).

2. Compute Gy (vertical gradient):

   (5+10) - (0+30) = 15 - 30 = -15
   (32+38) - (30+60) = 70 - 90 = -20
   (62+64) - (60+32) = 126 - 92 = 34
   

   Gy ≈ [-15, -20, 34].

3. Magnitude of Gradient (G): $G=\sqrt{G{x}^2+G{y}^2}$ For center pixel: $G=\sqrt{113^2+(-20{)}^2}=\sqrt{12769+400}=\sqrt{13169}\approx 114.75$.

4. Direction of Gradient (θ): $\theta ={\tan }^{-1}(Gy/Gx)$ For center pixel: $\theta ={\tan }^{-1}(-20/113)\approx -9.9°$ (almost horizontal edge).

Result: The center pixel has a strong horizontal edge (high magnitude, near 0° direction).

3. Region-Based Segmentation: Grouping Pixels

Region-based methods group homogeneous pixels (similar intensity, texture, or color) into regions. Approaches include:

• Thresholding: Split pixels into foreground/background based on a threshold (e.g., Otsu’s method).
• Splitting and Merging: Recursively divide/split regions until homogeneity is achieved.
• Growing Regions: Start with seed pixels and expand based on similarity.

Algorithm: Splitting and Merging

1. Initialization: Start with the entire image as one region.
2. Splitting: Divide regions into quadrants until a homogeneity criterion (e.g., variance < threshold) is met.
3. Merging: Combine adjacent regions if they are similar (e.g., mean intensity difference < threshold).

Example: Splitting a 4×4 Image

Initial Region (4×4):
| 50 | 52 | 60 | 65 |
| 51 | 53 | 61 | 66 |
| 49 | 50 | 59 | 64 |
| 48 | 49 | 58 | 63 |

Step 1: Split into 4 quadrants.

• Top-left (2×2):

  | 50 | 52 |
  | 51 | 53 |
  

  Variance = 2.5 (high → split further).
• Top-right (2×2):

  | 60 | 65 |
  | 61 | 66 |
  

  Variance = 6.25 (high → split further).
• Bottom-left (2×2):

  | 49 | 50 |
  | 48 | 49 |
  

  Variance = 1.25 (low → stop splitting).
• Bottom-right (2×2):

  | 59 | 64 |
  | 58 | 63 |
  

  Variance = 10 (high → split further).

Step 2: Merge adjacent regions if similar.

• Merge bottom-left (variance = 1.25) with its neighbors if their mean difference < 5.

4. Visual Perception in Segmentation

Humans segment images using Gestalt principles, which can guide algorithm design:

Application in OCR (Optical Character Recognition):

• Proximity: Groups letters in a word.
• Closure: Recognizes broken characters (e.g., "a" with a missing loop).

5. Distance Metrics for Region Connectivity

To measure how pixels are connected, we use distance metrics:

Example: Distance Between Pixels (3,4) and (6,8)

• Euclidean: $\sqrt{(6-3{)}^2+(8-4{)}^2}=\sqrt{9+16}=5$.
• Manhattan: $|6-3|+|8-4|=3+4=7$.

In the Real World

1. eSewa (Nepal) – Document Segmentation

   • Idea Used: Region-based segmentation (thresholding + connected components).
   • How: When you upload a receipt for payment, eSewa’s OCR system segments the text (amount, date, QR code) from the background using adaptive thresholding. This isolates each field for digit recognition.
   • Real Example: If you scan a faded receipt, segmentation ensures the "₹500" text is extracted cleanly, even if the background is noisy.

2. Pathao (Ride-Hailing) – Pedestrian Detection

   • Idea Used: Boundary-based segmentation (Canny edges + region growing).
   • How: Pathao’s autonomous safety features use edge detection to identify pedestrians at crosswalks. The Canny edge detector highlights human outlines, while region growing fills in gaps to form complete "pedestrian" regions.
   • Real Example: At a busy intersection in Kathmandu, the app’s camera segments a person waiting to cross, triggering a warning if the driver doesn’t slow down.

3. Nepal Police Traffic Monitoring – License Plate Extraction

   • Idea Used: Combination of thresholding (region-based) and edge detection (boundary-based).
   • How: Traffic cameras use segmentation to isolate license plates from vehicles. First, thresholding separates the plate (dark text on light background), then edge detection refines the plate’s boundaries for OCR.
   • Real Example: A traffic camera in Lalitpur captures a car’s plate, segments it from the vehicle’s body, and reads "3 KA 0123" despite shadows or dirt.

6. Comparison: Boundary vs. Region-Based Methods

Exam Tip

1. Define Key Terms Precisely:

   • Image segmentation: "Partitioning an image into non-overlapping, meaningful regions."
   • Boundary: "A curve that separates two regions with different properties."
   • Region: "A set of connected pixels with similar attributes."

2. For Numerical Questions:

   • Always show step-by-step calculations (e.g., Prewitt/Sobel gradients).
   • Use small matrices (3×3) for edge detection examples.
   • For splitting/merging, draw the image and label regions at each step.

3. Common Pitfalls:

   • Confusing edge detection with segmentation: Edges alone don’t segment an image; they must be linked into boundaries.
   • Ignoring homogeneity criteria: In splitting/merging, always state the threshold (e.g., "variance < 5").
   • Forgetting direction: In gradient-based methods, always compute both magnitude and direction.

4. Diagrams Are Your Friends:

   • Draw the image before and after segmentation.
   • Label regions in splitting/merging examples.
   • Sketch gradient vectors for edge detection questions.

5. Real-World Applications:

   • Link segmentation to medical imaging (tumor detection), autonomous vehicles (lane/pedestrian segmentation), or OCR (text extraction).
   • Example answer starter:

     "In eSewa’s receipt processing, region-based segmentation using Otsu’s thresholding isolates text fields, enabling accurate digit recognition even in low-light scans."

flowchart TD
    A["Image Input"] --> B["Boundary-Based\n(Prewitt/Sobel/Canny)"]
    A --> C["Region-Based\n(Thresholding/Splitting-Merging)"]
    B --> D["Edge Detection\n(Magnitude & Direction)"]
    C --> E["Homogeneity Check\n(Variance/Mean)"]
    D --> F["Link Edges into\nBoundaries"]
    E --> G["Group Pixels into\nRegions"]
    F --> H["Output: Segmented Image\n(Edges Only)"]
    G --> I["Output: Segmented Image\n(Regions Only)"]
    H & I --> J["Further Analysis\n(OCR, Object Recognition)"]