Data Quality Management: Ensuring Accuracy, Consistency & Trust

Why Data Quality Matters

Data is the lifeblood of modern businesses—from tracking patient records in hospitals to managing transactions in eSewa. But garbage in, garbage out (GIGO) is a harsh truth: poor data leads to wrong decisions, financial losses, and reputational damage. This unit teaches how to measure, monitor, and improve data quality to ensure reliability and trust.

1. Definitions & Core Concepts

What is Data Quality?

Data quality refers to the fitness of data for its intended use. High-quality data is:

• Accurate (correct and free from errors)
• Complete (no missing values)
• Consistent (same value across systems)
• Timely (up-to-date)
• Unique (no duplicates)
• Valid (conforms to business rules)

mindmap
  root((Data Quality))
    Accuracy
    Completeness
    Consistency
    Timeliness
    Uniqueness
    Validity

Key Terms

2. Dimensions of Data Quality

Data quality is evaluated across six dimensions:

1. Completeness

• Definition: No missing required fields.
• Example:
  • ❌ Patient record missing blood_type.
  • ✅ All fields (name, age, diagnosis) are filled.
• Impact: Incomplete data leads to wrong diagnoses (healthcare) or lost sales (e-commerce).

2. Accuracy

• Definition: Data matches real-world values.
• Example:
  • ❌ Bank account balance shows ₹10,000 but actual balance is ₹9,800.
  • ✅ Balance is ₹9,800 after reconciliation.
• Impact: Errors in NEPSE stock data can mislead investors.

3. Consistency

• Definition: Same data appears identical across systems.
• Example:
  • ❌ Customer’s address in Khalti is Kathmandu, Nepal but in eSewa is Kathmandu, Nepal (Updated).
  • ✅ Both systems show Kathmandu, Nepal.
• Impact: Duplicate records waste storage and cause confusion.

4. Timeliness

• Definition: Data is up-to-date.
• Example:
  • ❌ Flight status in Pathao shows "On Time" but is delayed by 2 hours.
  • ✅ Real-time updates from NTC/Ncell.
• Impact: Stock market crashes due to delayed data.

5. Uniqueness

• Definition: No duplicate records.
• Example:
  • ❌ Two patient records with the same NHID (National Health ID).
  • ✅ Each patient has a unique NHID.
• Impact: Double billing in hospitals.

6. Validity

• Definition: Data follows business rules.
• Example:
  • ❌ Phone number +977-98123456789 (invalid, should be 10 digits).
  • ✅ +977-9812345678 (valid).
• Impact: Failed transactions in eSewa/Khalti.

3. Data Quality Metrics

To measure data quality, we use key performance indicators (KPIs):

Example Calculation:

• A hospital has 500 patient records.
• 450 have complete diagnosis fields.
• Completeness Rate = (450/500) × 100 = 90%.

4. Causes of Poor Data Quality

Poor data quality arises from:

1. Human Errors

• Typing mistakes (e.g., Kathmandu vs Kathmandu City).
• Example: A Daraz seller enters ₹1,500 instead of ₹150.

2. System Issues

• Data entry forms with no validation.
• Example: Ncell allows invalid phone numbers (e.g., +977-98123456789).

3. Integration Problems

• Mismatched data formats between eSewa and banks.
• Example: Khalti sends transaction_id as TXN123 but bank expects TXN-123.

4. Lack of Standards

• No global ID for patients (e.g., NHID missing in rural hospitals).
• Example: NEPSE uses different ticker symbols for the same stock.

5. Data Aging

• Outdated records (e.g., NTC’s old customer database).
• Example: A Pathao driver’s license expired 2 years ago but DB still shows active.

5. Techniques to Improve Data Quality

1. Data Cleaning

• Process: Remove duplicates, correct errors, and fill missing values.
• Tools: SQL (UPDATE, DELETE), Excel (Remove Duplicates), Talend, Informatica.
• Example:

  -- Remove duplicate patients
  DELETE FROM patients
  WHERE id NOT IN (
      SELECT MIN(id) FROM patients
      GROUP BY name, dob
  );
  

2. Data Enrichment

• Process: Add missing or incomplete data from external sources.
• Example: Daraz enriches product descriptions with supplier data.

3. Data Validation

• Process: Ensure data follows business rules.
• Example:

  flowchart TD
      A["User enters phone: +977-9812345678"] --> B{"Is 10 digits?"}
      B -->|"Yes"| C["Accept"]
      B -->|"No"| D["Reject: Invalid format"]

4. Master Data Management (MDM)

• Process: Maintain a single source of truth for critical data (e.g., customer records).
• Example: Ncell uses MDM to sync customer data across NTC, Ncell, and eSewa.

5. Data Profiling

• Process: Analyze data to detect anomalies.
• Tools: IBM InfoSphere, SAS Data Quality.
• Example:

  flowchart TD
      A["Run data profiling"] --> B["Find missing ages"]
      B --> C["Flag records with NULL age"]
      C --> D["Send alert to admin"]

6. Automated Data Quality Tools

6. Real-World Applications

## In the real world

1. eSewa & Khalti (Payment Gateways)

   • Idea Used: Data Validation & Consistency
   • How?
     • Before processing a payment, eSewa checks:
       • Is the account number valid?
       • Does the amount match the merchant’s request?
       • Is the transaction timestamp recent?
     • If any check fails, the transaction is rejected (e.g., Invalid account error).
   • Worked Example:
     • A user tries to pay ₹1,000 but enters ₹1000.00 (extra decimal).
     • eSewa’s validation catches this and prompts correction.

2. Daraz (E-commerce)

   • Idea Used: Data Completeness & Timeliness
   • How?
     • Daraz ensures:
       • Product listings have images, prices, and stock levels.
       • Order status updates in real-time (e.g., "Shipped," "Delivered").
     • If a seller forgets to update stock, Daraz automatically marks "Out of Stock."
   • Worked Example:
     • A seller lists 100 units of a phone but only 50 are in stock.
     • Daraz’s inventory system flags this and reduces visible stock to 50.

3. NEPSE (Stock Exchange)

   • Idea Used: Data Accuracy & Timeliness
   • How?
     • NEPSE requires:
       • Real-time stock prices (no delays).
       • No duplicate trades (prevents fraud).
     • If a broker enters a duplicate trade, NEPSE’s system rejects it.
   • Worked Example:
     • A trader accidentally clicks "Buy" twice for the same stock.
     • NEPSE’s data validation detects the duplicate and only processes one trade.

7. Challenges in Data Quality Management

8. Best Practices for Data Quality

1. Define Quality Standards Early

   • Example: Ncell sets rules like:
     • Phone numbers must be 10 digits.
     • Customer names must be ≤50 characters.

2. Automate Data Validation

   • Example: Khalti uses regex to check:

     /^\+977-\d{10}$/  // Validates Nepali phone numbers
     

3. Regular Data Audits

   • Example: NEPSE runs weekly checks for:
     • Duplicate trades.
     • Incorrect stock symbols.

4. Train Employees

   • Example: Daraz sellers are trained to:
     • Enter correct product details.
     • Update stock levels daily.

5. Use Data Governance Policies

   • Example: eSewa has:
     • Ownership rules (who can edit transaction data?).
     • Access controls (only admins can delete records).

9. Case Study: Improving Data Quality in a Hospital

Scenario: A hospital’s patient database has:

• 20% missing diagnoses.
• 5% duplicate records.
• 10% incorrect ages.

Steps to Fix:

1. Data Cleaning:

   • Run SQL to fill missing diagnoses with default "Pending".
   • Delete duplicates using:

     DELETE FROM patients WHERE id NOT IN (
         SELECT MIN(id) FROM patients GROUP BY name, dob
     );
     

2. Data Validation:

   • Add a rule: Age must be between 0 and 120.
   • Flag incorrect ages (e.g., age = 150).

3. Automated Monitoring:

   • Set up alerts for:
     • New duplicates.
     • Missing fields.

Result:

• Completeness: 95% (up from 80%).
• Uniqueness: 100% (no duplicates).
• Accuracy: 99% (ages corrected).

Exam Tip

• Focus on:
  • Definitions (e.g., What is data completeness?).
  • Techniques (e.g., How does data profiling work?).
  • Real-world examples (e.g., How does eSewa validate transactions?).
• Common Exam Questions:
  • "Explain three dimensions of data quality with examples."
  • "How would you improve data quality in a hospital management system?"
  • "Compare data cleaning and data enrichment."
• Marking Scheme:
  • Definitions: 1-2 marks each.
  • Examples: 2-3 marks.
  • Diagrams/flows: 3-4 marks (e.g., data validation flowchart).
  • Case study analysis: 5-6 marks.