Skip to main content

@remotion/whisper-webgpuv4.0.518

Transcribe audio locally in the browser using timestamped Whisper models and WebGPU through Transformers.js, and convert the result to @remotion/captions.

Installation​

npx remotion add @remotion/whisper-webgpu

Example​

transcribe.ts
import {clearStaleModels, resampleTo16Khz, toCaptions, transcribe} from '@remotion/whisper-webgpu';

await clearStaleModels();

const file = new File([], 'audio.wav');
const channelWaveform = await resampleTo16Khz({file});

const transcription = await transcribe({
  channelWaveform,
  model: 'small.en',
  language: 'en',
});

const {captions} = toCaptions({whisperWebGpuOutput: transcription});
console.log(captions);

The model is downloaded on the first call and cached by the browser. Call loadWhisperModel() first to display model download progress.

APIs​

canUseWhisperWebGpu()
Check whether WebGPU is available
getAvailableModels()
List timestamped models and their download sizes
clearStaleModels()
Remove models discontinued by newer versions
isWhisperModelCached()
Check whether a model is downloaded
loadWhisperModel()
Download and initialize a model
transcribe()
Transcribe a waveform with word-level timestamps
toCaptions()
Convert a transcription to Remotion captions
resampleTo16Khz()
Decode and resample browser audio
disposeWhisperModel()
Release model memory

Requirements​

WebGPU requires HTTPS in production or localhost during development. Use canUseWhisperWebGpu() to check for a usable adapter.

License​

MIT