NLP: Ambiguities, Steps, Turing Test & Applications

TAKEAWAYS:

• NLP bridges human language and machine understanding via tokenization, parsing, and semantic analysis, but faces lexical, syntactic, and semantic ambiguities.
• The Turing Test requires machines to mimic human conversation convincingly, testing contextual and pragmatic understanding.
• Real-world NLP powers apps like eSewa’s chatbots (intent recognition), Khalti’s fraud detection (sentiment analysis), and Google Translate (machine translation).
• Ambiguity resolution relies on context, statistical models, or rule-based systems (e.g., disambiguating "bank" in a financial vs. river context).
• NLP pipelines (tokenization → POS tagging → parsing → semantics) are visualized as flowcharts, while search spaces (e.g., parsing trees) show how machines explore possible interpretations.

1. Introduction to Natural Language Processing (NLP)

NLP is a subfield of AI that enables machines to understand, interpret, and generate human language. It combines linguistics, computer science, and machine learning to process text/speech. Key applications include:

• Chatbots (e.g., eSewa’s customer support),
• Machine translation (Google Translate),
• Sentiment analysis (Khalti’s customer feedback).

Why NLP Matters in Nepal?

• eSewa: Uses NLP to classify user queries (e.g., "bill payment" vs. "complaint") via intent recognition.
• Ncell’s IVR: Resolves ambiguities in voice commands (e.g., "call mom" vs. "call mother").
• NEPSE stock analysis: Extracts trends from news headlines using named entity recognition (NER).

2. Steps in NLP (Pipeline Visualization)

NLP processes text through modular stages, often visualized as a pipeline:

flowchart LR
    A["Input Text: 'The bank is high.'"] --> B["Tokenization"]
    B --> C["Part-of-Speech Tagging"]
    C --> D["Parsing (Syntax Tree)"]
    D --> E["Semantic Analysis"]
    E --> F["Output: Contextual Meaning"]

Key Steps Explained

3. Ambiguities in NLP (With Real-World Examples)

Ambiguities arise when words/sentences have multiple valid interpretations. Types include:

A. Lexical Ambiguity (Word-Level)

• Same word, different meanings.
  • Example: "Bank" (river vs. financial institution).
  • Real-world tie-in: Ncell’s IVR must disambiguate "bank" in "Call the bank" (user might mean Nabil Bank or a river!).

B. Syntactic Ambiguity (Sentence Structure)

• Same words, different tree structures.
  • Example: "Visiting relatives can be tedious."
    • Tree 1: "Visiting" (verb) → relatives are being visited.
    • Tree 2: "Relatives" (subject) → visiting them is tedious.
  • Visualization:

    graph TD
        A["Visiting relatives can be tedious."]
        A --> B["[Visiting [relatives]] can be tedious."]
        A --> C["Visiting [relatives can be tedious]."]

  • Worked Example: For the sentence "The police chased the thief on the bicycle.", parse both possibilities:
    1. Police chased (thief on a bicycle).
    2. Police (on a bicycle) chased the thief.

C. Semantic Ambiguity (Meaning-Level)

• Logically valid but context-dependent meanings.
  • Example: "I saw the man on the hill with a telescope."
    • Who had the telescope? The speaker or the man?
  • Real-world tie-in: Pathao’s ride-hailing app must resolve:
    • "Pick me up at the hotel." → Which hotel? (User’s location vs. nearest hotel.)

D. Pragmatic Ambiguity (Implied Meaning)

• What’s not said matters.
  • Example: "Can you pass the salt?" (Request vs. question about ability).
  • Real-world tie-in: Khalti’s customer support uses NLP to detect sarcasm in complaints like:
    • "Great! My transaction failed again." → Negative sentiment, not literal praise.

4. The Turing Test and NLP

Proposed by Alan Turing, the Imitation Game tests if a machine can fool a human into thinking it’s another human. NLP is critical because:

• Machine must understand context, idioms, and sarcasm.
• Requires pragmatic competence (e.g., handling indirect requests).

Properties to Pass the Turing Test

5. Real-World NLP Applications in Nepal

A. eSewa: Intent Recognition

• Problem: Users type "pay bill" or "check balance" ambiguously.
• NLP Solution:
  1. Tokenization: Split input into ["pay", "bill", "1234"].
  2. Intent Classification: Match to "payment" intent.
  3. Entity Extraction: "1234" → bill reference number.
• Visualization:

  flowchart LR
      A["User Input: 'Pay bill 1234'"] --> B["Tokenize"]
      B --> C["Intent: PAYMENT"]
      C --> D["Entity: BILL_ID=1234"]
      D --> E["Action: Process Payment"]

B. Khalti: Fraud Detection via Sentiment Analysis

• Problem: Detect fake reviews (e.g., "Product is amazing!!!" but 1-star rating).
• NLP Solution:
  • Sentiment Score: "Amazing!!!" → Positive (but mismatched with rating).
  • Flag for review: Trigger manual verification.

C. NTC: Customer Complaint Routing

• Problem: Users complain about "slow internet" or "no signal."
• NLP Solution:
  • Keyword Extraction: "slow" + "internet" → Route to network team.
  • Entity Recognition: "Area: Kathmandu-3" → Assign to local technician.

6. Worked Example: Disambiguating "Fly"

Sentence: "The fly caught the bug." Ambiguity: Is "fly" a noun (insect) or verb (past tense of "to fly")?

Step-by-Step Resolution

1. Tokenization: ["The", "fly", "caught", "the", "bug."]
2. POS Tagging:
   • "fly" → Could be noun or verb.
3. Parsing Trees:
   • Option 1 (Noun): [S [NP The fly] [VP caught [NP the bug]]]
     • Meaning: The insect "fly" caught the bug.
   • Option 2 (Verb): [S [NP The] [VP fly caught [NP the bug]]]
     • Meaning: The subject "The" flew and caught the bug (unlikely; "The" is not a noun here).
4. Semantic Check:
   • "The fly" (insect) is more plausible than "The" as a subject.
5. Final Interpretation: "The insect fly caught the bug."

Real-world tie-in: Pathao’s dispatch system might misroute a driver if it misinterprets:

• "The fly to the airport." → Verb (go) vs. noun (insect).

7. NLP Techniques for Ambiguity Resolution

8. Challenges in NLP for Nepali Language

• Limited Datasets: Few annotated Nepali corpora for training.
• Morphological Complexity: Nepali has sandhi (word fusion) and case markers.
  • Example: "मेरो" (my) + "किताब" (book) → "मेरो किताब" (my book).
• Code-Switching: Mixing Nepali and English (e.g., "Khalti ma transfer garnu parxa?").
• Solution: Use transfer learning (e.g., fine-tuning multilingual BERT).

Exam Tip

1. Ambiguities: Always explain lexical, syntactic, and semantic ambiguities with examples (e.g., "bank," "fly").
2. NLP Pipeline: Draw a flowchart for tokenization → parsing → semantics.
3. Turing Test: List 5 properties (e.g., natural language, context memory) and link to NLP.
4. Real-World Applications: Relate to eSewa, Khalti, or Ncell in answers (examiners love local examples!).
5. Worked Examples: Show step-by-step parsing for ambiguous sentences (e.g., "Visiting relatives...").
6. Visuals: Include syntax trees or NLP pipelines in answers to score extra marks.

9. Summary Table: NLP Ambiguities vs. Solutions

10. Practice Questions (Exam-Style)

1. Short Answer:

   • Define lexical ambiguity with an example from Nepali (e.g., "घर" as house or home).
   • List 3 properties a machine must have to pass the Turing Test.

2. Long Answer:

   • Explain the NLP pipeline with a diagram and trace how eSewa processes the input: "Check my bill status for order ID 12345."
   • Discuss two ambiguities in the sentence: "The cat saw the bird with the telescope." Show parse trees for both interpretations.

3. Application:

   • How does Khalti use NLP to detect fraudulent transactions? Explain with a flowchart of their NLP system.