$0 Multilingual Speech Recognition in Your Chatbot with Cloudflare Whisper

If you're building a chatbot and want to add voice input without paying for Google Speech-to-Text, AWS Transcribe, or OpenAI Whisper API, you already have everything you need — inside Cloudflare's free tier.

In this article, I'll walk you through exactly how I integrated Cloudflare Workers AI Whisper large-v3-turbo into the AI chatbot on my portfolio site, supporting 99 languages at absolutely zero cost. I'll also share the two production bugs I ran into — and precisely how I fixed them.

Why Cloudflare Workers AI?

Cloudflare offers speech recognition as part of Workers AI, which runs on Cloudflare's global edge network. The free tier includes:

• Workers AI inference: Free up to the daily limits (generous enough for personal projects and small apps)
• Zero cold starts: Runs at the edge, closest to your user
• No API keys to manage: Bound directly to your Worker via env.AI
• 99-language support: Including Chinese, Japanese, Korean, French, Spanish, Arabic, and more

The best part: you don't need to host a model, manage GPU instances, or deal with third-party API rate limits. It just works.

Whisper Base vs Whisper Large v3 Turbo: An Honest Comparison

Cloudflare Workers AI offers two Whisper model variants. Choosing the right one matters.

@cf/openai/whisper (Base)

This is the original Whisper base model (≈39M parameters).

Pros:

• Fast inference — best for latency-sensitive applications
• Lower resource usage per request
• Battle-tested and stable on Cloudflare

Cons:

• Higher Word Error Rate (WER), especially on non-English languages
• Struggles with accents, technical vocabulary, and mixed-language input
• Audio input format: number[] (Uint8Array spread into a plain JS array)

Best for: High-volume apps where speed matters more than accuracy, or English-only use cases.

@cf/openai/whisper-large-v3-turbo (Turbo)

This is a distilled version of Whisper large-v3 (≈809M parameters), optimized to be ~2× faster than large-v3 while retaining most of its accuracy.

Pros:

• ~50% lower WER compared to base — dramatically better on Chinese, Japanese, Korean, etc.
• Excellent multilingual accuracy even with accents
• Supports optional parameters: language, task (transcribe/translate), vad_filter, beam_size
• Returns rich output: text, word_count, segments with WebVTT timing

Cons:

• Heavier than base — slightly higher latency per request
• Audio input format: Base64 string (not a number array — this caused my first bug)
• Does not stably decode webm/opus containers (this caused my second bug)

Best for: Multilingual chatbots, voice assistants, accessibility tools, transcription pipelines where accuracy is critical.

Side-by-Side Summary

| | @cf/openai/whisper | @cf/openai/whisper-large-v3-turbo | |---|---|---| | Parameters | ~39M | ~809M | | Languages | 99 | 99 | | Accuracy | Good | Excellent | | Speed | Fastest | Fast (2× faster than large-v3) | | audio input format | number[] | Base64 string | | Supported containers | webm, wav, mp3 | wav, mp3 (webm unstable) | | Extra output fields | text only | text, word_count, segments | | Cost on Cloudflare | Free tier | Free tier |

Architecture

Here's the full pipeline I built:

Browser Microphone
    ↓
MediaRecorder (records webm/opus — default browser format)
    ↓ [client-side transcoding]
AudioContext.decodeAudioData() → decodes webm/opus
    ↓
encodeWAV() → 16-bit PCM WAV (mono, native sample rate)
    ↓ HTTP POST multipart/form-data
/api/transcribe  (Next.js Edge Route on Cloudflare Pages)
    ↓
getRequestContext().env.AI  (Cloudflare AI binding)
    ↓
btoa() → Base64 encode in chunks of 8192 bytes
    ↓
env.AI.run('@cf/openai/whisper-large-v3-turbo', { audio: base64 })
    ↓
{ text: "transcribed result" } → fills chatbot input

Step 1: Bind the AI Binding in Wrangler

In your wrangler.jsonc (or wrangler.toml), add the AI binding:

{
  "ai": {
    "binding": "AI"
  }
}

Then declare the type in env.d.ts:

interface CloudflareEnv {
  AI: any;
}

Run npx wrangler types to generate accurate types from your live bindings.

Step 2: The Server-Side Transcription Route

Create src/app/api/transcribe/route.ts. The two critical requirements:

1. export const runtime = 'edge' — mandatory for getRequestContext() to work on Cloudflare Pages
2. Base64 encode the audio — turbo model requires this; Edge Runtime has no Buffer

import { NextRequest, NextResponse } from 'next/server';
import { getRequestContext } from '@cloudflare/next-on-pages';

export const runtime = 'edge';

export async function POST(req: NextRequest) {
  try {
    const formData = await req.formData();
    const file = formData.get('file') as File;

    if (!file) {
      return NextResponse.json({ error: 'No audio file provided' }, { status: 400 });
    }

    const { env } = getRequestContext();

    if (!env.AI) {
      return NextResponse.json({ error: 'Cloudflare AI binding is missing' }, { status: 500 });
    }

    const arrayBuffer = await file.arrayBuffer();
    const audioBytes = new Uint8Array(arrayBuffer);

    // Edge Runtime has no Buffer — chunk-encode to Base64 to avoid stack overflow
    let binary = '';
    for (let i = 0; i < audioBytes.length; i += 8192) {
      binary += String.fromCharCode(...audioBytes.subarray(i, i + 8192));
    }
    const audioBase64 = btoa(binary);

    const response = await env.AI.run('@cf/openai/whisper-large-v3-turbo', {
      audio: audioBase64,
    });

    if (!response || !response.text) {
      throw new Error('No transcription returned from AI model');
    }

    return NextResponse.json({ text: response.text });
  } catch (err: unknown) {
    console.error('Transcription API Error:', err);
    return NextResponse.json(
      { error: err instanceof Error ? err.message : String(err) },
      { status: 500 }
    );
  }
}

Step 3: Client-Side Recording with WAV Transcoding

The browser's MediaRecorder API records in webm/opus by default — which turbo cannot reliably decode (error 3030). The fix: transcode to WAV in the browser before uploading.

The WAV Encoder

function encodeWAV(audioBuffer: AudioBuffer): Blob {
  const numChannels = 1; // mono is sufficient for speech
  const sampleRate = audioBuffer.sampleRate;
  const samples = audioBuffer.getChannelData(0);
  const buffer = new ArrayBuffer(44 + samples.length * 2);
  const view = new DataView(buffer);

  const writeString = (offset: number, str: string) => {
    for (let i = 0; i < str.length; i++) view.setUint8(offset + i, str.charCodeAt(i));
  };

  // RIFF/WAVE header
  writeString(0, 'RIFF');
  view.setUint32(4, 36 + samples.length * 2, true);
  writeString(8, 'WAVE');
  writeString(12, 'fmt ');
  view.setUint32(16, 16, true);
  view.setUint16(20, 1, true);  // PCM
  view.setUint16(22, numChannels, true);
  view.setUint32(24, sampleRate, true);
  view.setUint32(28, sampleRate * numChannels * 2, true);
  view.setUint16(32, numChannels * 2, true);
  view.setUint16(34, 16, true); // 16-bit
  writeString(36, 'data');
  view.setUint32(40, samples.length * 2, true);

  // float32 → int16 PCM
  let offset = 44;
  for (let i = 0; i < samples.length; i++) {
    const s = Math.max(-1, Math.min(1, samples[i]));
    view.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7fff, true);
    offset += 2;
  }

  return new Blob([view], { type: 'audio/wav' });
}

The Recording Logic

mediaRecorder.onstop = async () => {
  const webmBlob = new Blob(audioChunksRef.current, { type: 'audio/webm' });
  stream.getTracks().forEach((track) => track.stop());

  // Guard: empty recordings (accidental taps) → silently ignore
  if (webmBlob.size < 1000) return;

  setIsLoading(true);
  try {
    const arrayBuffer = await webmBlob.arrayBuffer();
    const audioCtx = new AudioContext();
    const decoded = await audioCtx.decodeAudioData(arrayBuffer);
    await audioCtx.close(); // always close to avoid "too many AudioContexts"
    const wavBlob = encodeWAV(decoded);

    const formData = new FormData();
    formData.append('file', wavBlob, 'recording.wav');

    const response = await fetch('/api/transcribe', { method: 'POST', body: formData });
    // ... handle response
  } catch (error) {
    // ...
  } finally {
    setIsLoading(false);
  }
};

Step 4: Dual-Mode Mic Button (Hold-to-Talk + Tap-to-Toggle)

Using Pointer Events (works for both mouse and touch), I detect the press duration to support both interaction patterns:

const TAP_THRESHOLD_MS = 400;

const handleMicPressStart = async (e: React.PointerEvent) => {
  e.preventDefault();
  if (isLoading) return;
  if (recordModeRef.current === 'toggle') { stopListening(); return; }
  if (recordModeRef.current !== 'idle') return;

  pressStartRef.current = Date.now();
  recordModeRef.current = 'pending';
  await startListening();
};

const handleMicPressEnd = (e: React.PointerEvent) => {
  e.preventDefault();
  if (recordModeRef.current !== 'pending') return;

  const elapsed = Date.now() - pressStartRef.current;
  if (elapsed < TAP_THRESHOLD_MS) {
    recordModeRef.current = 'toggle'; // tap → keep recording, wait for second tap
  } else {
    stopListening(); // hold → stop immediately on release
  }
};

Two Bugs I Hit in Production

Bug 1: Error 5006 — Type Mismatch

After switching from @cf/openai/whisper to @cf/openai/whisper-large-v3-turbo, I got error 5006.

Root cause: I kept the old audio format audio: [...audioBytes] (a number array), which is what the base model expects. The turbo model expects a Base64 string. These are completely different input schemas.

Fix: Convert Uint8Array → Base64 using btoa() in chunks on the server side.

Bug 2: Error 3030 — Failed to Decode Audio File

After fixing the Base64 format, I hit error 3030.

Root cause: The browser records in webm/opus. Even though the Base64 encoding was now correct, the turbo model couldn't decode the webm/opus container. Cloudflare's Whisper turbo only reliably handles WAV and MP3.

Fix: Decode the webm/opus recording with AudioContext.decodeAudioData() in the browser, then re-encode as 16-bit PCM WAV before uploading. WAV is the most universally supported Whisper input format.

Key Gotchas & Tips

1. export const runtime = 'edge' is mandatory. Without it, getRequestContext() returns nothing on Cloudflare Pages.

2. No Buffer in Edge Runtime. When encoding to Base64, you cannot use Buffer.from(bytes).toString('base64'). Use btoa() with chunking to avoid call stack overflow on large audio files.

3. The turbo and base models have different audio field types. This is not clearly documented — I found it through trial, error, and reading the actual error codes.

4. Always close AudioContext. Browsers cap the number of simultaneous AudioContext instances. Always await audioCtx.close() after decoding.

5. Guard against empty recordings. If the user taps the mic accidentally and releases immediately, decodeAudioData will throw on an empty blob. Check blob.size < 1000 and return early.

6. Base64 adds ~33% overhead. For a 10-second voice recording (~160KB raw PCM), the Base64 payload is ~215KB — well within Cloudflare's limits. Still, keep recordings short for best UX.

7. iOS requires a user gesture to create AudioContext. Using onPointerDown (not a timer or useEffect) satisfies this requirement.

The Result

After implementing this pipeline, my chatbot supports:

• Hold-to-talk: Press and hold the mic button while speaking, release to transcribe
• Tap-to-toggle: Tap once to start, say your message, tap again to transcribe
• 99 languages with excellent accuracy on Chinese, Japanese, Korean, English, and more
• Zero cost — fully within Cloudflare's free tier
• No third-party API keys — everything runs on the Cloudflare edge

If you're building a multilingual AI product and want voice input without paying for it, Cloudflare Workers AI Whisper is the answer. The two bugs I hit are non-obvious from the documentation — hopefully this article saves you hours of debugging.

The full implementation is live on elvisli.ca. All source code is available on GitHub.