Why do speech recognition systems struggle with Malayalam?

Mostly training data. Speech models are typically trained on read-aloud text, which is slower, more clearly articulated and grammatically written rather than spoken. That gap is much wider for Malayalam than for English, which has far more genuinely conversational training audio available to close it.

What is Manglish and why does it break voice AI?

Manglish is the everyday mixing of Malayalam and English inside a single sentence, which is the normal register for phone conversation in Kerala. It breaks most multilingual systems because they detect one language per utterance before processing, and a code-switched sentence has no single correct answer to that question.

Does translating Malayalam to English first solve it?

No, it fails on exactly the same sentences. Translation requires deciding the source language first, which is the assumption code-switching breaks. It also corrupts proper nouns — an English treatment or product name inside a Malayalam sentence gets translated into a different word entirely.

Can voice AI handle Malayalam accurately enough for business calls?

For booking, enquiry handling and message-taking, yes, provided it is trained on real conversational audio rather than read-aloud corpora and does not run a language-detection or translation step in front of the pipeline. The important design property is that it asks when unsure instead of committing to a plausible mistake.

Why voice AI mishears Malayalam

29 August 2026

Every speech system handles Malayalam badly in roughly the same way, and it is not because Malayalam is unusually difficult. It is because of what the training audio is, and because of a design decision almost everyone makes about multilingual speech that is wrong for the way people here actually talk.

The training data problem

To train a speech model you need audio paired with an accurate transcript, in bulk. The cheapest way to get that at scale is to have people read written text aloud. You already have the transcript; you just need someone to say it.

This produces a lot of usable data and one large blind spot. Read-aloud speech is slower than conversation, more fully articulated, free of interruption, and grammatically written rather than spoken. A model trained on it learns a register that exists mainly in the recording studio.

Then it meets a phone call. Somebody talking at normal speed over a mobile connection, cutting themselves off, restarting the sentence, with a television on behind them. The gap between the two is much wider for Malayalam than for English, because English has vastly more genuinely conversational training audio in circulation to close it.

Code-switching is not an edge case

Listen to a booking call in Kerala. The greeting is Malayalam, the day is English, the time is Malayalam, the price is English. Not because the speaker is showing off, but because that is the shape the language takes here. Manglish is the register, not a deviation from it.

This breaks the standard architecture for multilingual speech, which is to detect the language first and then run the model for that language. Language detection assumes there is one answer per utterance. Here there is not one answer per sentence, let alone per call.

What happens in practice is that the detector picks a language, and everything in the other one gets forced through the wrong model. English words in a Malayalam sentence come out as Malayalam words that sound similar and mean nothing. The transcript is grammatical and wrong.

The translation-layer trap

The next fix people reach for is a translation layer. Take the Malayalam, translate to English, do the reasoning in English where the models are strongest, translate the answer back.

It is a sensible design and it fails on the same sentences. To translate, you must first decide what the source language is — the exact assumption that just broke. Feed it a sentence that is half English and it has to pick, and half the meaning goes through the wrong path.

Proper nouns take the worst of it. A caller says a treatment name, a model number or a place in English inside a Malayalam sentence, and the translation step helpfully converts it into a different word. The booking is now against something the customer never asked for, and nobody finds out until they arrive.

What actually helps

Three things, none of them clever.

Train on real calls rather than read audio, accepting that this data is far more expensive to assemble because somebody has to transcribe it honestly, including the parts where the caller mumbles.

Stop detecting the language before processing. Handle the utterance in the form it arrived in, mixed, rather than resolving it to one language first.

Protect the spans that must survive verbatim — numbers, names, times, places. These are where a plausible error is most expensive, because a wrong appointment time reads exactly like a right one until the customer turns up.

Real call audio over read-aloud corpora, including the noisy and the mumbled.
No language-detection step in front of the pipeline.
No round-trip translation for code-switched speech.
Verbatim handling for numbers, names, times and places.
Accent coverage across the state, not one city.

The honest limits

None of this makes the problem disappear. Heavy background noise still degrades accuracy, an unfamiliar proper noun is still a coin flip, and a caller who mumbles a phone number will still occasionally be misheard — a human receptionist asks them to repeat it, and so should the system.

The realistic goal is not perfection. It is that when the system is unsure, it asks, rather than committing confidently to a plausible mistake. Most of the damage from voice AI on a business line is not caused by failure to understand. It is caused by misunderstanding without noticing.