Pilot Study & Research Validity: Design, Errors & Real-World Checks

TAKEAWAYS:

• A pilot study is a mini-version of your main research to identify flaws in tools, methods, or logistics before full-scale data collection.
• Research validity (internal, external, construct, face) measures whether your study answers the right question correctly and reliably.
• Common errors in research design (e.g., sampling bias, measurement ambiguity) can invalidate results—always pre-test with a pilot.
• Real-world tie: Daraz uses pilot tests for new delivery routes (e.g., testing a "same-day delivery" model in Kathmandu-3 before rolling it out nationwide).
• Exam focus: Define pilot study, list validity types, and explain how to fix errors (e.g., using Cronbach’s alpha for reliability).
• Case study: Nabil Bank’s customer satisfaction surveys failed in 2022 due to low construct validity (questions didn’t measure "satisfaction" clearly)—they fixed it by pilot-testing questions with a small branch sample.

1. What Is a Pilot Study?

A pilot study is a small-scale trial run of your research methods to:

• Test questionnaires, interviews, or observation tools for clarity/ambiguity.
• Estimate time/costs for full data collection.
• Identify logistical issues (e.g., hard-to-reach respondents).

How It Works: Step-by-Step

flowchart TD
    A["Start: Define Research Goals"] --> B["Step 1: Draft Tools\n(Questionnaire, Interview Guide)"]
    B --> C["Step 2: Select Pilot Sample\n(5–10% of target population)"]
    C --> D["Step 3: Conduct Trial\n(Collect data, observe issues)"]
    D --> E["Step 4: Analyze Feedback\n- Are questions clear?\n- Is data usable?\n- Are respondents cooperative?"]
    E --> F["Step 5: Revise Tools\nFix ambiguities, adjust sampling"]
    F --> G["Step 6: Proceed to Full Study\nOR Iterate Pilot"]

Worked Example: eSewa’s User Feedback Pilot

Problem: eSewa wanted to add a "quick-reply" feature for customer complaints but needed to test survey questions first. Pilot Steps:

1. Drafted 10 questions (e.g., "How satisfied are you with our response time?" on a 1–5 scale).
2. Tested with 50 users in Lalitpur.
3. Issues found:
   • "Response time" was ambiguous (users interpreted it as call wait time vs. email reply time).
   • 5-point scale caused confusion (some users circled two numbers).
4. Fix: Simplified to a 3-point scale ("Fast," "Average," "Slow") and defined "response time" as "time to first acknowledgment."

Outcome: The revised survey was used in a full study, revealing 68% of users wanted faster replies—a key insight for eSewa’s 2023 app update.

2. Why Pilot Studies Matter

3. Research Validity: The 4 Key Types

Validity ensures your research answers the right question correctly. The four types:

Visual: Validity Hierarchy

mindmap
  root((Research Validity))
    Internal
      "Controls confounds"
      "Example: Random assignment in experiments"
    External
      "Generalizability"
      "Example: Representative sampling"
    Construct
      "Measures the *right* concept"
      "Example: Pilot-testing survey items"
    Face
      "Appears valid to stakeholders"
      "Example: Expert review of questions"

4. Common Errors in Research Design (And How Pilots Fix Them)

5. Case Study: How Ncell Used a Pilot to Improve Validity

Problem: Ncell’s 2021 "customer loyalty" survey had low construct validity—questions like "Do you recommend Ncell?" (Net Promoter Score) didn’t distinguish between service quality and price. Pilot Steps:

1. Drafted 15 questions (mixed loyalty, speed, and price).
2. Tested with 200 users in Birgunj and Dharan.
3. Found:
   • Users ignored price-related questions (they assumed Ncell was cheap).
   • "Recommend" was influenced by data speed, not loyalty.
4. Revised:
   • Split into 3 scales:
     • Service quality (e.g., "How often do calls drop?").
     • Price fairness (e.g., "Is the plan value for money?").
     • Loyalty (e.g., "Would you switch if a competitor offered 10% off?").
5. Result: The full survey showed service quality (not price) drove loyalty—Ncell then focused on network upgrades.

6. How to Write a Pilot Study Report (Exam Tip)

Examiners often ask for the structure of a pilot study report. Use this template:

flowchart TD
    A["Title Page"] --> B["Abstract\n(Brief goals, methods, key findings)"]
    B --> C["Introduction\n- Research problem\n- Pilot objectives"]
    C --> D["Methodology\n- Tools used (e.g., questionnaire)\n- Sample details\n- Data collection process"]
    D --> E["Findings\n- What worked?\n- What didn’t?\n- Quantitative/qualitative feedback"]
    E --> F["Discussion\n- How findings inform main study\n- Revisions made"]
    F --> G["Conclusion\n- Lessons learned\n- Recommendations for full study"]

Example Abstract for a Pilot:

"This pilot study tested a 20-item questionnaire on ‘student satisfaction with online learning’ (TU, 2023) with 50 BBS 4th-semester students. Key issues included ambiguous Likert scales (e.g., ‘often’ vs. ‘sometimes’) and low response rates from evening shifts. Revisions included a 5-point scale and targeted sampling during daytime. These changes are critical for the full study’s validity."

## In the Real World

1. eSewa’s Feature Testing

   • Idea: Pilot studies for new user interfaces.
   • How: Before launching "eSewa Pay Later," they tested the checkout flow with 100 users in Bhaktapur. Found that 30% abandoned carts due to unclear interest rates—fixed by adding a pop-up calculator.

2. Daraz’s Delivery Route Optimization

   • Idea: External validity (can findings apply nationwide?).
   • How: Piloted a "same-day delivery" model in Kathmandu-3 with 500 orders. Discovered traffic jams on Thursdays invalidated their "2-hour delivery" claim—now they adjust routes dynamically.

3. Nabil Bank’s Loan Default Prediction

   • Idea: Construct validity (does the model measure risk correctly?).
   • How: A pilot with 200 loan applicants found their initial model used income level alone—ignoring employment stability. After revision, the model’s accuracy improved from 65% to 82%.

## Exam Tip

1. Define pilot study in 1 sentence:

   "A small-scale trial of research methods to identify flaws in tools, sampling, or logistics before full data collection."

2. Validity types: Memorize the 4 types with one example each (use the table above).

3. Common errors: Expect questions on sampling bias, measurement error, or low reliability. Always suggest a pilot fix (e.g., "Pre-test questions with a stratified sample to check for ambiguity").

4. Case study approach: If asked "How would you improve X study?", structure your answer as:

   • Identify the error (e.g., "low construct validity").
   • Propose a pilot fix (e.g., "test questions with a small group").
   • Real-world tie (e.g., "Like Ncell’s 2021 survey").

5. Avoid these mistakes:

   • ❌ Saying a pilot is the "final study."
   • ❌ Ignoring reliability (always mention Cronbach’s alpha or test-retest).
   • ❌ Assuming convenience samples are valid for external validity.

Final Visual: Pilot Study vs. Full Study