Data Sources, Cleaning Techniques & Quality Assurance

Core Concepts

Data Sources: Where Data Comes From

Data can be broadly classified into structured (tabular, fixed schema) and unstructured (text, images, videos) formats. Nepalese companies like eSewa (structured transaction logs) and Pathao (unstructured ride logs with GPS coordinates) rely on both types.

mindmap
  root((Data Sources))
    Structured
      Databases["SQL (MySQL, PostgreSQL)"]
      Spreadsheets["Excel, CSV"]
      APIs["REST, GraphQL"]
    Unstructured
      Text["PDFs, Logs"]
      Multimedia["Images, Videos"]
      Social["WhatsApp, Facebook"]
    Semi-Structured
      JSON["API responses"]
      XML["Config files"]

Data Collection Methods

1. Web Scraping

• Extracts data from websites (e.g., Nepal Stock Exchange (NEPSE) price scrapers).
• Tools: BeautifulSoup, Scrapy.
• Example: Scraping Daraz product reviews to analyze sentiment.

2. APIs (Application Programming Interfaces)

• Structured data access (e.g., Khalti’s payment API for transaction records).
• Example: Fetching NTC’s electricity consumption data via API.

3. Databases

• Direct queries (e.g., Global IME Bank’s loan data).
• Example: SQL query to extract Pathao driver earnings from a PostgreSQL database.

-- Example SQL query for Pathao driver data
SELECT driver_id, SUM(fare) AS total_earnings
FROM rides
WHERE date = '2023-10-01'
GROUP BY driver_id;

4. Sensors/IoT

• Real-time data (e.g., NTC’s smart meters for electricity usage).
• Example: Kathmandu traffic sensors feeding into Pathao’s route optimization.

Data Cleaning: Fixing Dirty Data

Common Issues & Solutions

Handling Missing Data

1. Deletion: Remove rows/columns (if <5% missing).
   • Example: Dropping Ncell customer records with missing phone numbers.
2. Imputation:
   • Mean/Median: For numerical data (e.g., NEPSE closing prices).
   • Mode: For categorical data (e.g., Daraz product categories).
3. Advanced: KNN imputation (for complex datasets like bank loan defaults).

# Example: Imputing missing NEPSE prices
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy='mean')
cleaned_prices = imputer.fit_transform(nepse_data[['price']])

Data Quality Metrics

In the Real World

1. eSewa: Uses APIs to collect transaction data and cleans it to detect fraud (e.g., duplicate payments).
2. Pathao: Relies on GPS sensor data cleaned via outlier removal to optimize routes in Kathmandu traffic.
3. NEPSE: Scrapes stock prices from websites, cleans missing values, and ensures consistency for traders.

Worked Example:

• Problem: Daraz’s order queue has missing delivery times.
• Solution:
  1. Impute missing times with median delivery duration per city.
  2. Remove outliers (e.g., 3-day deliveries in Kathmandu).
  3. Validate with Khalti payment timestamps for consistency.

Exam Tip

• Focus Areas:
  • Differentiate between structured/unstructured data (e.g., SQL vs. web logs).
  • Know 3 data cleaning techniques (e.g., imputation, deduplication).
  • Relate metrics to Nepalese companies (e.g., Khalti’s accuracy, NTC’s completeness).
• Common Pitfalls:
  • Forgetting to validate data after cleaning.
  • Overusing deletion for missing data (use imputation instead).
• Scoring Boosters:
  • Draw a data cleaning workflow in exams.
  • Compare API vs. scraping with real examples (e.g., Khalti API vs. NEPSE scraper).

Visual Summary:

flowchart TD
    A["Data Collection"] --> B["Web Scraping"]
    A --> C["APIs"]
    A --> D["Databases"]
    A --> E["Sensors"]
    B --> F["Clean Missing Values"]
    C --> F
    D --> G["Remove Duplicates"]
    E --> H["Filter Outliers"]
    F --> I["Validate"]
    G --> I
    H --> I
    I --> J["Analyze"]