Speech Processing & Synthesis: Synthesis, Generation, and Real-World Speech Tech

TAKEAWAYS:

• Speech synthesis converts text-to-speech (TTS) using phonemes, prosody, and acoustic models, while speech generation involves real-time voice production from audio signals.
• Concatenative synthesis (stitching pre-recorded speech segments) and parametric synthesis (synthesizing speech from parameters) are two primary techniques, each with trade-offs.
• Acoustic models (e.g., Hidden Markov Models, neural networks) map text to phonemes and then to waveforms, while prosody (pitch, rhythm) is critical for natural speech.
• Real-world applications include eSewa’s voice authentication, Pathao’s voice-based ride booking, and Ncell’s IVR systems, where speech processing reduces friction in user interactions.
• Challenges include naturalness (robotic vs. human-like speech), language support (limited for low-resource languages), and real-time processing for live applications.
• Speech recognition (e.g., Google Assistant) and synthesis (e.g., Siri) are complementary but distinct; synthesis focuses on generating speech from text or parameters, not interpreting it.

1. Introduction to Speech Processing and Synthesis

Speech processing involves analyzing, synthesizing, and understanding human speech. It is divided into two main areas:

• Speech Synthesis: Generating speech from non-speech inputs (e.g., text, phonemes, or parameters).
• Speech Generation: Converting audio signals into speech (e.g., voice cloning, voice conversion).

Speech synthesis is widely used in:

• Automated call centers (e.g., Ncell’s IVR systems).
• Voice assistants (e.g., Google Assistant, Siri).
• Accessibility tools (e.g., screen readers for visually impaired users).
• E-commerce (e.g., Daraz’s voice-based order tracking).

2. Speech Synthesis Techniques

Speech synthesis can be broadly classified into two categories:

1. Concatenative Synthesis
2. Parametric Synthesis

2.1 Concatenative Synthesis

Concatenative synthesis works by stitching together pre-recorded speech segments to form new utterances. It is widely used because it produces highly natural-sounding speech.

How it works:

1. A speech database is created with recorded phonemes, syllables, or words.
2. During synthesis, the system selects and concatenates the most appropriate segments based on the input text.
3. Prosody (pitch, rhythm, stress) is adjusted to make the output sound natural.

Advantages:

• Highly natural-sounding speech.
• Works well for small vocabularies.

Disadvantages:

• Limited flexibility (cannot generate new sounds not in the database).
• Large storage requirements for the speech database.

Example:

• eSewa’s voice verification system uses concatenative synthesis to generate voice prompts for user authentication. The system concatenates pre-recorded segments to create unique verification phrases, reducing fraud.

2.2 Parametric Synthesis

Parametric synthesis generates speech by synthesizing it from parameters (e.g., formant frequencies, pitch, duration) rather than using pre-recorded segments. It is more flexible but often sounds less natural.

How it works:

1. The input text is converted into phonemes (basic speech sounds).
2. An acoustic model maps phonemes to articulatory parameters (e.g., lip shape, tongue position).
3. A voice synthesizer (e.g., a vocoder or source-filter model) generates the speech waveform from these parameters.

Types of Parametric Synthesis:

Advantages:

• Highly flexible (can generate new sounds not in a database).
• Smaller storage requirements.

Disadvantages:

• Often sounds robotic or unnatural.
• Requires complex acoustic models.

Example:

• Pathao’s voice-based ride booking uses parametric synthesis to generate dynamic voice responses. The system converts text instructions (e.g., "Your driver is 2 minutes away") into speech using a neural TTS model, ensuring real-time responsiveness.

3. Acoustic Models in Speech Synthesis

Acoustic models map phonemes (input) to speech waveforms (output). They are the backbone of modern speech synthesis systems.

3.1 Hidden Markov Models (HMMs)

• How it works:

  • Models speech as a sequence of hidden states (phonemes) with observable outputs (spectral features).
  • Uses Gaussian Mixture Models (GMMs) to represent the probability distribution of spectral features for each state.

• Advantages:

  • Works well for small datasets.
  • Statistically robust.

• Disadvantages:

  • Struggles with complex prosody (e.g., emotional speech).
  • Requires large labeled datasets.

Example:

• Early Ncell IVR systems used HMM-based TTS to generate voice prompts for customer service. While not as natural as modern systems, it was sufficient for basic interactions.

3.2 Neural Network-Based Models

Modern systems use deep learning (e.g., RNNs, Transformers) to improve naturalness and prosody.

Types:

1. Recurrent Neural Networks (RNNs)

   • Capture temporal dependencies in speech.
   • Example: WaveNet (Google) generates high-quality speech but is computationally expensive.

2. Transformers (e.g., Tacotron 2)

   • Use self-attention mechanisms to model long-range dependencies.
   • Example: Amazon Polly uses a Transformer-based model for natural-sounding TTS.

Advantages:

• Highly natural speech.
• Better prosody control (e.g., emotional speech).

Disadvantages:

• Requires large datasets and significant computational resources.
• Training is complex and time-consuming.

Example:

• Google Assistant uses a Transformer-based TTS model to generate speech for voice responses. The system analyzes the user’s intent and context to produce emotionally appropriate and natural-sounding replies.

4. Prosody in Speech Synthesis

Prosody refers to the melodic, rhythmic, and stress patterns in speech that make it sound natural. Key components include:

• Pitch (Fundamental Frequency, F0): Determines how high or low the voice sounds.
• Duration: How long a sound is held.
• Intensity: Loudness of the speech.

How prosody is controlled:

1. Rule-based systems: Apply predefined rules (e.g., "question words get higher pitch").
2. Data-driven systems: Learn prosody from recorded speech (e.g., neural networks).
3. Hybrid systems: Combine rules and data-driven approaches.

Example:

• eSewa’s voice verification adjusts prosody to ensure the generated voice sounds like a human. For instance, it adds slight variations in pitch and duration to mimic natural speech patterns, reducing the chance of voice spoofing.

5. Challenges in Speech Synthesis

Despite advancements, speech synthesis faces several challenges:

6. Speech Generation vs. Speech Synthesis

While often used interchangeably, speech generation and speech synthesis are distinct concepts:

Example:

• NEPSE’s automated trading alerts use speech synthesis to convert text updates (e.g., "Stock X rose by 2%") into voice announcements.
• Voice cloning apps (e.g., ElevenLabs) use speech generation to replicate a person’s voice from a small audio sample.

7. Real-World Applications

7.1 eSewa’s Voice Authentication

• Idea Used: Concatenative synthesis + prosody adjustment.
• How it Works:
  • eSewa generates unique voice verification phrases by concatenating pre-recorded segments.
  • Prosody is adjusted to sound natural and reduce fraud attempts.
• Worked Example:
  • User hears: "Your verification code is 1-2-3-4. Please repeat after me: 'The sky is blue.'"
  • The system concatenates recorded digits and phrases to create the prompt dynamically.

7.2 Pathao’s Voice-Based Ride Booking

• Idea Used: Neural TTS for real-time voice responses.
• How it Works:
  • Pathao’s app uses a Transformer-based TTS model to generate voice updates (e.g., "Your driver is arriving in 1 minute").
  • The system analyzes the user’s location and ride status to produce context-aware responses.
• Worked Example:
  • User asks: "How long until my ride arrives?"
  • Pathao’s system generates: "Your driver is 3 minutes away. Estimated arrival: 10:15 AM."

7.3 Ncell’s IVR Systems

• Idea Used: HMM-based TTS for basic voice prompts.
• How it Works:
  • Ncell’s IVR uses HMMs to generate voice menus (e.g., "Press 1 for balance inquiry").
  • While not as natural as neural TTS, it is sufficient for high-volume, low-complexity interactions.
• Worked Example:
  • User dials 123# and hears: "Welcome to Ncell. Press 1 for balance, 2 for recharge."
  • The system concatenates pre-recorded digits and phrases to form the menu.

8. Exam Tip

This unit is highly visual and application-driven, so focus on:

1. Comparing synthesis techniques (concatenative vs. parametric) with a table (as shown above).
2. Explaining acoustic models (HMMs vs. neural networks) and their pros/cons.
3. Tying real-world examples (eSewa, Pathao, Ncell) to synthesis techniques.
4. Addressing challenges (naturalness, language support) with concrete examples.
5. Distinguishing speech synthesis from speech generation clearly.

Common Pitfalls:

• Confusing speech synthesis (text-to-speech) with speech recognition (speech-to-text).
• Forgetting to mention prosody when discussing naturalness.
• Not linking examples to specific techniques (e.g., saying "eSewa uses TTS" without specifying concatenative or parametric).

How to Score Full Marks:

• Use Mermaid diagrams for synthesis techniques (e.g., flowchart of concatenative vs. parametric).
• Include real labelled images of:
  • A vocoder block diagram (for parametric synthesis).
  • A neural TTS architecture (e.g., Tacotron 2).
  • A prosody waveform (showing pitch and duration).
• For worked examples, trace the flow (e.g., "Input text → phonemes → acoustic model → waveform → prosody adjustment → output").

flowchart TD
    A["Input Text"] --> B["Text-to-Phoneme Conversion"]
    B --> C["Phoneme-to-Acoustic Parameters"]
    C --> D["Acoustic Model (HMM/Neural Network)"]
    D --> E["Waveform Generation"]
    E --> F["Prosody Adjustment"]
    F --> G["Output Speech"]
    subgraph Concatenative Synthesis
        H["Speech Database"] -->|"Select Segments"| I["Concatenate Segments"]
        I --> F
    end
    subgraph Parametric Synthesis
        J["Vocoder/Neural TTS"] --> E
    end

Figure 1: Flowchart of speech synthesis techniques (concatenative vs. parametric). The left path shows how concatenative synthesis selects and stitches pre-recorded segments, while the right path illustrates parametric synthesis generating speech from parameters.

stateDiagram-v2
    [*] --> TextInput
    TextInput --> PhonemeConversion
    PhonemeConversion --> AcousticModel
    AcousticModel --> WaveformGeneration
    WaveformGeneration --> ProsodyAdjustment
    ProsodyAdjustment --> OutputSpeech
    OutputSpeech --> [*]

Figure 2: State diagram of the speech synthesis pipeline. Each state represents a step from input text to output speech, highlighting the role of acoustic models and prosody.

Figure 3: Comparison table of speech synthesis techniques. Neural TTS strikes a balance between naturalness and flexibility, making it ideal for modern applications.