Knowva AI Language Platform
An interactive language learning platform designed to assess user proficiency and generate custom curriculum paths for up to 500k users. Combines OpenAI Whisper (WER = 0.232) for speech assessment, Google Gemini (rating 4.64/5) for personalized tutoring, and xTTS + WAV2Lip (0.92 Speaker Similarity) for generating lip-synced audio-visual training avatars.
Problem Statement
Conventional language learning applications rely on static worksheets and pre-recorded audio, failing to adapt to a user's unique spoken accent, grammar level, or interactive communication pace. To scale level assessments for over 500k active users, systems must evaluate spoken speech in real-time, generate custom learning material dynamically, and synthesize lifelike video instructors that enhance comprehension and pronunciation.
Solution
Built the Knowva Platform, an AI-powered language tutor. Developed an adaptive proficiency examiner backend utilizing Google Gemini to adjust question difficulties dynamically. Integrated OpenAI's Whisper to transcribe voice answers and calculate Word Error Rate (WER) to assess pronunciation. Created an audio-visual generation pipeline utilizing xTTS for voice synthesis and WAV2Lip to sync the audio with a photorealistic tutor avatar, providing users with immersive face-to-face feedback.
Architecture
Knowva utilizes a microservices architecture. The transcription engine handles incoming voice chunks, the LLM engine manages curriculum generation and conversational agents, and the TTS/avatar engine synthesizes speech and lip-synced video streams.
Key Features
- ▸Adaptive level placement exams powered by Gemini LLMs
- ▸Speech transcription and Word Error Rate (WER) scoring using Whisper
- ▸Multilingual text-to-speech engine using xTTS with high speaker similarity cloning
- ▸Audio-to-video lip-synchronization using pre-trained WAV2Lip models
- ▸Personalized curriculum generation based on user performance profiles
- ▸Real-time speech-to-text validation and pronunciation analysis
Challenges
- ⚡Synchronizing synthesized audio with avatar videos in a low-latency environment, requiring optimized model quantization and memory management.
- ⚡Achieving high accuracy in speech recognition for non-native speakers, solved by fine-tuning Whisper thresholding and audio pre-processing.
- ⚡Designing prompts for Gemini that maintain a consistent grading scale and prevent vocabulary drift.
Results & Metrics
Developed an architecture designed to scale proficiency testing to 500k users
Achieved low transcription word error rate (WER = 0.232) under diverse accent conditions
Attained a high speaker resemblance index (0.92 Cosine Similarity) for cloned tutor voices
Secured an average user rating of 4.64/5 for personalized custom learning materials
Lessons Learned
- 💡Audio quality (noise reduction and volume normalization) dramatically affects Whisper transcription success, and adding a WebRTC noise gate is crucial.
- 💡Combining xTTS and WAV2Lip requires precise frame-rate aligning (25fps vs 16kHz audio sample chunks) to avoid visible lip drift.
- 💡Caching prompt skeletons and vector embeddings of learning materials reduces API costs and response latency by over 40%.
Case Study Overview
Case Study: Engineering Knowva Platform
Building a scalable, interactive language tutor requires combining multiple specialized AI models into a cohesive, low-latency execution pipeline. Knowva achieves this by coupling Speech-to-Text (STT), Large Language Models (LLMs), Text-to-Speech (TTS), and Lip-Synchronization technologies to build an adaptive, human-like teaching experience.
Technical Architecture & Flow
The system processes student inputs through an interactive cycle: transcribing spoken responses, evaluating accuracy, deciding next steps, and rendering video and voice feedback.
┌────────────────────────────────────────┐
│ Student Speaks │
└──────────────────┬─────────────────────┘
│ (Microphone Audio)
▼
┌────────────────────────────────────────┐
│ Speech-to-Text (OpenAI Whisper) │ ◄── Calc WER (0.232)
└──────────────────┬─────────────────────┘
│ (Transcribed Text)
▼
┌────────────────────────────────────────┐
│ Adaptive Evaluation (Google Gemini) │ ◄── Custom Curriculum
└──────────────────┬─────────────────────┘
│ (Tutor Text Response)
▼
┌────────────────────────────────────────┐
│ Voice Synthesis (xTTS v2) │ ◄── Speaker Similarity (0.92)
└──────────────────┬─────────────────────┘
│ (Synthesized Audio)
▼
┌────────────────────────────────────────┐
│ Lip-Sync Generation (WAV2Lip) │ ◄── Video Frame Sync
└──────────────────┬─────────────────────┘
│ (Avatar Video Stream)
▼
┌────────────────────────────────────────┐
│ Student Watches Video │
└────────────────────────────────────────┘Core Model Implementations
1. Pronunciation Assessment via Speech-to-Text (STT)
- Whisper Integration: OpenAI's Whisper model transcribes non-native user speech. Speech preprocessing includes
librosa-based silent chunk trimming and volume normalization to handle user microphone variations. - WER Scoring: Implemented a Levenshtein-distance algorithm to calculate Word Error Rate (WER) against expected pronunciation guidelines. Knowva achieved a WER = 0.232 under various accents.
2. Adaptive Examination & LLM Optimization
- Adaptive Testing: Uses Google Gemini to dynamically adjust question difficulty based on historical response accuracy.
- Structured Output: Prompts are constrained to return valid JSON schemas mapping rating scores, detected grammatical mistakes, and next-step recommendations.
- User Rating: The generated learning material and conversational interactions secured a user satisfaction score of 4.64 / 5.
3. Voice Cloning & Lip-Sync Video Avatars
- Voice Synthesis: Uses the
xTTS v2model to perform high-similarity speaker cloning (achieving 0.92 Cosine Speaker Similarity), letting students choose personalized voices for their virtual tutor. - Lip Synchronization: Feeds synthesized audio chunks and a reference face image/video into a pre-trained
WAV2Lipmodel. The generated video displays perfectly synced lips, significantly improving the student's listening comprehension.
Performance & Model Specs
To compare the key metrics across the pipeline:
Technologies
Gallery
Links
Related Projects
Arabic Speech-to-Text & LLM NER Pipeline
Developed an end-to-end Arabic dialect transcription, text preprocessing, and structured Named Entity Recognition (NER) pipeline using Whisper, MARBERT, and LangChain.
RAG-Based Mental Health Chatbot
An AI-driven mental health support chatbot leveraging multi-stage NLP classification, Qdrant hybrid search, and cross-encoder reranking.