Project Description

Case Studies / AI & ML

AI & ML

Phoneme Level Pronunciation Assessment for a Major Language Learning Platform

Our client is a leading language learning platform based in North America that caters to a global audience.

Book a strategy call
Phoneme Level Pronunciation Assessment for a Major Language Learning Platform
12%increase in user engagement
ClientLeading language learning platform based in North America that caters to a global audience
PracticeAI & ML

The Client

Client Overview

Our client is a leading language learning platform based in North America that caters to a global audience. The platform focuses on enhancing language proficiency, with a special emphasis on improving pronunciation skills. The client envisions a user-friendly interface where individuals can see a word on the screen, record their pronunciation, and receive a detailed performance assessment.

01

Challenges

Out-of-the-box pronunciation scoring APIs from GCP, Azure, and AWS could not provide the level of detail we wanted to have while displaying pronunciation scoring. We were able to achieve this by building our own model using an open-source framework.

02

Approach

Technical implementation
  • Kaldi- an open-source Automatic Speech Recognition (ASR) toolkit is used that contains various recipes for training customized acoustic models
  • Labeled Audio Data (recordings and spoken words) and Pronunciation Lexicon (words and corresponding sequences of phonemes) are collected from Librispeech dataset of English language audio files
  • MFCC (Mel-frequency Cepstral Coefficients), CMVN (Cepstral mean and variance normalization), and i-vectors audio features are extracted
  • The model is trained using TDNN (Time Delay Neural Network) on an NVIDIA A10 GPU instance for cost-effectiveness, parallel execution, and speed.
  • The Deep Neural Network (DNN) framework is built using the GOP_Speechocean recipe from Kaldi, the acoustic model, and the language model.
  • Trained acoustic model performs phoneme detection on new audio data outputting the most likely sequence of phonemes.
  • The model is evaluated against Word Error Rate (WER)
  • 'compute-gop' script calculates GOP, flags mispronunciations below a threshold, and displays results with antilog conversion for human readability.
Download LabeledTraining Data Data Preparationand LexiconCreation Extract MFCC, CMVNand i-vectorFeatures Acoustic ModelTraining Evaluate the ModelUsing WER Need MoreData? Fine TuningRequired? Fine Tune ModelParameters Add More Data andRe-train the Model Publish the Model NOYES YESNO Training flowRetraining loop
Figure 1 — Model training on an NVIDIA A10 GPU.
Download LabeledTraining Data Data Preparationand LexiconCreation Extract MFCC, CMVNand i-vectorFeatures Acoustic ModelTraining Evaluate the ModelUsing WER Need MoreData? Fine TuningRequired? Fine Tune ModelParameters Add More Data andRe-train the Model Publish the Model NOYES YESNO Training flowRetraining loop ×
PHONEME DETECTION AND GOP CALCULATION Audio InputFrom User API(Gunicorn + Nginx) Data Preparation Extract MFCC, CMVN,I-vector Features Compute GOP Convert to HumanReadable Format Display Output
Figure 2 — Phoneme detection and GOP calculation, running on a Linux EC2 instance.
PHONEME DETECTION AND GOP CALCULATION Audio InputFrom User API(Gunicorn + Nginx) Data Preparation Extract MFCC, CMVN,I-vector Features Compute GOP Convert to HumanReadable Format Display Output ×

03

Impact

12%

Increased user engagement by 12% as the solution enriched the user experience.

8%

8% surge in user retention with a more engaging and effective language learning platform.

10%

10% rise in user referrals and testimonials, driving positive word-of-mouth recommendations.

Want results like these from your own data?

Talk to a specialist

See how you could get similar results

Rudder Analytics builds data and AI systems that ship — and stay tied to accuracy, cost, and customer experience.

Book a strategy callSee more case studies