Handling Transcription Code-Switching in Desktop Recording

The short answer

Code-switching is when a speaker alternates between two or more languages inside one conversation, one turn, or one sentence. It’s where most transcription pipelines break down, with error rates running roughly 5–10x higher than for audio in just one language. When building desktop recorders to handle multilingual audio, teams should make sure to capture clean, isolated audio and choose a desktop recording SDK that allows them to swap transcription providers without changing the architecture.

What code-switching actually looks like in a recorded meeting

Three patterns matter, and they get progressively harder for a transcription pipeline.

Turn-level alternation. Speaker A talks in English, Speaker B answers in Italian. The language boundary lines up with a speaker boundary (since each speaker uses a different language), so a system with working diarization has a natural unit to work with.

Inter-sentential switching. The same speaker finishes an English sentence and starts the next one in Italian. Speaker changes are no longer a usable signal, but sentence- or segment-level language labels are usually enough to accurately transcribe the speech.

Intra-sentential switching. The language changes inside the sentence: "Let's ship the integration on Friday porque necesitamos cerrar el trimestre." The system cannot assign one language to the utterance. It has to recognize both, in sequence, and preserve both.

Intra-sentential switching is where production pipelines break, and it is also the most common pattern in fluent bilingual speech. Worse, the languages can mix inside a single word (for example, a German or Arabic speaker might attach native morphology to an English stem).

The PIER study measured this separately and found intra-word switching substantially harder than inter-word: on the German–English DECM set, Whisper-large-v3 scored 45.23 PIER on intra-word cases versus 30.42 on inter-word. On Arabic–English ArzEn, intra-word PIER reached 105.71 — above 100% because the model inserts more wrong words than there are reference words to match.

How the code-switching failures actually manifest

Five distinct failure modes show up in code-switched recordings, each with different downstream consequences.

Substitution into the dominant language. A short Spanish phrase inside English audio gets rewritten as similar-sounding English. "Necesito cancelar my subscription" becomes "necessity can sell my subscription." The transcript is fluent, but confidently incorrect.

Silent deletion. The embedded phrase is dropped entirely. This is the most dangerous situation for compliance and clinical use cases, because the absence leaves no trace to be audited.

Unrequested translation. This happens when the model translates instead of transcribing. HuggingFace's June 2026 benchmark of seven frontier ASR systems found that Whisper Large V3 Turbo, called without an explicit language parameter on code-switched audio, defaults to translating into English rather than preserving the language spoken. Its WER on that benchmark ranged from 0.16 to 0.61 across four language pairs, with degradation relative to monolingual English peaking at +0.85 on German–English.

Lost switch boundaries. The words come back roughly right, but the API returns one language label for the whole recording. This makes it difficult for applications that need to understand what languages were spoken and when.

Error clustering on the embedded language. HuggingFace’s benchmark also produced a counterintuitive result. Across all models and all four language pairs tested, errors concentrated on the English portions of code-switched utterances, despite English being the language these models typically handle best. The likely explanation is that English words or phrases within bilingual professional speech disproportionately carry technical vocabulary and product names. Which is to say: the errors land on the highest-value parts of the transcript.

Their regression analysis added two more useful findings. The number of language switches in an utterance was the strongest predictor of whether an error occurred at all. The Code-Mixing Index (how densely the two languages are interwoven) predicted how significant the error would be when it happened.

What a code-switching-ready desktop pipeline requires

Practical requirements, in rough order of how often teams miss them:

Clean capture before anything else. The audio capture pipeline needs to be high-quality in order for transcripts to pick up the speech accurately.
Language metadata at the granularity your product needs. To locate a switch inside a sentence, you need language labels at the word level.
The ability to swap providers without re-architecting. Performance varies enormously by language pair. There is no single best provider for multilingual audio, so the best provider for one pair of languages may not be the best for another.
Evaluation on your own production audio. Run a benchmark that has real accents, overlapping speech, and the specific language pairs your users actually speak.

General speech-to-text benchmarks show how a model performs on a published test set. They do not necessarily show how a provider will handle the languages, switching patterns, speakers, and recording conditions in your product. Many also do not let developers test their own audio and ground truth.

Why use the Desktop Recording SDK for multilingual desktop recording

To transcribe multilingual audio accurately, teams building desktop recorders must have the right architecture. Most teams shipping desktop recorders integrate a recording SDK rather than owning the stack. Recall.ai's Desktop Recording SDK is the option we point people to most often, for the following reasons:

The Desktop Recording SDK offers high-quality audio capture. Audio capture has a significant impact on transcript quality. If the audio being captured has degraded quality, then it will be even more difficult to transcribe multilingual audio correctly.

Desktop recorders built with native macOS or Windows APIs struggle with many audio capture issues. One issue is audio isolation – figuring out a way to not record audio on the device that’s unrelated to the meeting. Another common thing developers need to account for is echo cancellation, where audio from the device gets picked up by the microphone and is recorded twice. Both of these problems can have a huge impact on the quality of the transcripts generated, and will make it much harder to handle multilingual audio correctly.

The Desktop Recording SDK lets you switch transcription providers without rebuilding architecture. The same API surfaces Recall.ai Transcription, ElevenLabs, Deepgram, AssemblyAI, AWS Transcribe, Rev, and Speechmatics. Many of the providers offer word-level language labels and are specifically tuned for code-switched audio. If benchmarking shows that a different provider wins on your dominant language pair, you can easily change your configuration without needing to rebuild your integration.

None of this makes code-switching a solved problem, but it does help teams spend their time effectively. Teams using a recording SDK can spend their effort choosing and evaluating transcription providers against their real audio, which is the work that actually helps generate more accurate transcripts.

How to test your own pipeline

Recall.ai maintains an open-source code-switching benchmark that runs the same multilingual recordings through multiple providers and scores transcript accuracy, language-identification accuracy, switch detection, diarization status, and latency against labeled ground truth.

Build the test set from audio your product actually receives. Include monolingual controls in each language, turn-level alternation, inter-sentential switching, and intra-sentential switching. Add short embedded phrases, borrowed words, names, acronyms, and product terms — the categories most likely to cause issues with code-switching. Then score per language, not just in aggregate (This is important – for example, a mostly-English recording will post a respectable overall WER even if the provider missed every Spanish sentence in it). By the end, this should show you the best transcription provider for your actual use case.

FAQ

Why is code-switching difficult for ASR models? It removes an assumption most systems rely on: that nearby words are likely to be in the same language. Acoustic models are typically trained on one language at a time, as natural code-switched speech is difficult to collect and train models on. The transition point between languages is exactly where the model has seen almost no training examples, so it assigns very low probability to the correct continuation and tends to "repair" the foreign word into a phonetically similar native one.

How much worse is code-switched transcription? Roughly 5–10x, depending on the pair and the model. Whisper-large-v3 scores 29.43% WER on Fisher Spanish–English code-switched audio against 4.91% on monolingual Spanish. Measured only on the switched words, PIER climbs to 36.41% on the same data and 58.73% on SEAME Mandarin–English.

Does language detection solve code-switching? No. Language detection identifies which language is being spoken; code-switching support means handling more than one within the same recording. Systems that run detection first and route to a monolingual recognizer work for language changes at clean boundaries and fail on mid-sentence switches, because the routing decision has to be made before there is enough context to make it.

Should I use real-time or async transcription for multilingual desktop recording? Async is generally more accurate, and the gap widens on code-switched audio because the model can use surrounding context. Use real-time for live UI and async for the durable record if your product needs both.

Can post-processing fix code-switched transcripts? Only partially. An LLM cleanup pass can repair minor errors, but it cannot recover words the acoustic model never detected. Deletions and unrequested translations are unrecoverable after the fact, since there is nothing in the text indicating what was lost.