NLP & Speech AI

Arabic Speech-to-Text & LLM NER Pipeline

A comprehensive speech-to-text and NLP pipeline designed for academic thesis research. Transcribes Egyptian Arabic audio/video recordings using a fine-tuned Whisper model, aligns timestamps, preprocesses text with MARBERT tokenizers, and utilizes LangChain and GPT-4o-mini to extract structured named entities.

PythonOpenAI WhisperLangChainGPT-4o-miniPyTorchOpenCVMediaPipeMARBERTNLTKPandas

Problem Statement

Transcribing and extracting semantic information from dialectal Arabic (such as Egyptian Arabic) is highly challenging. Standard ASR models struggle with local slang, while conventional NER tools fail on colloquial, unstructured text transcripts. Furthermore, academic research on multi-modal datasets requires precise timestamp alignment and auxiliary video feature tracking.

Solution

Built an end-to-end Python pipeline. Implemented audio extraction, resampling (to 16kHz), and segmentation using PyDub. Transcribed colloquial speech using WafaaFraih's fine-tuned Whisper model. Cleaned and tokenized the transcripts with NLTK and the MARBERT encoder. Constructed a LangChain pipeline with gpt-4o-mini and a Pydantic structured output parser to extract named entities with high fidelity. Integrated MediaPipe hand and skeleton landmark tracking to synchronize visual gestures with transcribed audio cues.

Architecture

The system processes media files (e.g. MP4) by splitting audio into overlapping chunks for Whisper transcription and utilizing OpenCV to extract video frames. Transcribed Arabic texts are aligned with exact start/end timestamps and saved in tabular formats. The aligned text is passed to a LangChain extraction chain, utilizing a structured Pydantic schema to extract named entities. Concurrently, MediaPipe processes the video stream to track pose/hand movements.

Key Features

  • ▸Egyptian Arabic audio/video transcription using fine-tuned Whisper
  • ▸Precise subtitle/excel timestamping alignment at millisecond levels
  • ▸Dialectal text tokenization and stop-word filtering using MARBERT
  • ▸Structured named entity extraction using LangChain and Pydantic validation
  • ▸Computer vision gesture tracking with MediaPipe landmark detection

Challenges

  • ⚡High variance in dialectal Arabic pronounciation causing transcription errors. Solved by utilizing a specialized Egyptian Arabic fine-tuned Whisper model and preprocessing the audio signals (resampling, normalisation).
  • ⚡Ensuring the LLM returns consistently formatted named entities. Solved by implementing LangChain's PydanticOutputParser with fallback handlers to prevent JSON formatting violations.

Results & Metrics

✓

Achieved highly accurate, dialect-aware transcription of Egyptian Arabic recordings.

✓

Automated structured Named Entity Recognition (NER) with 100% schema validation pass rates.

✓

Aligned multi-modal gesture tracking data with corresponding transcription timestamps for thesis research.

Lessons Learned

  • 💡Colloquial Arabic datasets require specialized dialect models (like MARBERT and dialect-tuned Whisper); MSA (Modern Standard Arabic) models perform poorly on local dialects.
  • 💡Structured output parsers are essential for downstream data analysis pipelines to prevent malformed API responses from crashing automated scripts.

Case Study Overview

Project Overview

This project implements a multi-modal data processing pipeline developed to transcribe, analyze, and structure spoken Egyptian Arabic for academic thesis research.

By combining advanced speech-to-text models, deep learning dialect encoders, LLM-driven structured extraction, and computer vision skeletonization, the pipeline transforms raw audio-visual content into high-fidelity research-ready datasets.

Technical Architecture & Multi-Modal Pipeline

Egyptian Arabic Speech-to-NER & Gesture Pipeline Architecture
        ┌──────────────────────────────────────────────────────────┐
        │             Audio-Visual Raw Egyptian Video              │
        └────────────────────────────┬─────────────────────────────┘
                                     │
                    ┌────────────────┴────────────────┐
                    ▼                                 ▼
        ┌───────────────────────┐         ┌─────────────────────────┐
        │ Acoustic Preprocess   │         │ OpenCV Frame Extractor  │
        │  (16kHz Resampling)   │         │  (Hand & Body Tracking) │
        └───────────┬───────────┘         └────────────┬────────────┘
                    │                                  │
                    ▼                                  ▼
        ┌───────────────────────┐         ┌─────────────────────────┐
        │ Whisper Egyptian ASR  │         │  MediaPipe Pose Engine  │
        │ (Millisecond Timings) │         │ (Spatial Gesture Vector)│
        └───────────┬───────────┘         └────────────┬────────────┘
                    │                                  │
                    ▼                                  │
        ┌───────────────────────┐                      │
        │ MARBERT Tokenizer &   │                      │
        │ Stopword Filtering    │                      │
        └───────────┬───────────┘                      │
                    │                                  │
                    ▼                                  │
        ┌───────────────────────┐                      │
        │  gpt-4o-mini &        │                      │
        │ Pydantic NER Parser   │                      │
        └───────────┬───────────┘                      │
                    │                                  │
                    └────────────────┬─────────────────┘
                                     │ (Multi-Modal Alignment)
                                     ▼
        ┌──────────────────────────────────────────────────────────┐
        │      Thesis Research Dataset (Entities + Gestures)       │
        └──────────────────────────────────────────────────────────┘

Processing Pipeline Stages

  1. Acoustic Preprocessing & Ingestion:
    • Extract audio tracks from video source files.
    • Segment audio files using PyDub and resample waveforms to 16kHz for Whisper compatibility.
  2. ASR Dialect Transcription:
    • Run inference using the specialized Egyptian Arabic speech recognition model WafaaFraih/whisper-small-egyptian-arabic-All.
    • Export structured transcripts complete with millisecond start and end timestamps.
  3. Arabic Dialect NLP Tokenization:
    • Tokenize text using the UBC-NLP/MARBERT dialect encoder.
    • Clean punctuation and filter out dialect-specific stopwords.
  4. LLM Named Entity Extraction:
    • Pass transcripts through a LangChain prompt template.
    • Query gpt-4o-mini with strict output instructions.
    • Parse results into a structured JSON array of named entities using LangChain's PydanticOutputParser.
  5. Computer Vision Feature Tracking:
    • Extract frame-by-frame image data with OpenCV.
    • Track hand joints and body posture using MediaPipe to align spatial gestures with verbal entities.

Technologies

PythonOpenAI WhisperLangChainGPT-4o-miniPyTorchOpenCVMediaPipeMARBERTNLTKPandas

Links

Related Projects

NLP & Speech AI

Knowva AI Language Platform

An AI-driven language learning platform featuring adaptive proficiency exams, real-time Whisper STT, Gemini-powered content customization, and WAV2Lip lip-sync.

NLP & Speech AI

RAG-Based Mental Health Chatbot

An AI-driven mental health support chatbot leveraging multi-stage NLP classification, Qdrant hybrid search, and cross-encoder reranking.