Skip to main content

Extracting frames from a video in JavaScript

Extracting frames from a video file, for example to display a filmstrip in an editing interface, can be done using Mediabunny.

Here's a extractFrames() function you can use coyp and paste into your project:

extract-frames.ts
ts
import {ALL_FORMATS, Input, InputDisposedError, UrlSource, VideoSampleSink} from 'mediabunny';
 
type Options = {
  track: {width: number; height: number};
  container: string;
  durationInSeconds: number | null;
};
 
export type ExtractFramesTimestampsInSecondsFn = (options: Options) => Promise<number[]> | number[];
 
export type ExtractFramesProps = {
  src: string;
  timestampsInSeconds: number[] | ExtractFramesTimestampsInSecondsFn;
  onFrame: (frame: VideoFrame) => void;
  signal?: AbortSignal;
};
 
export async function extractFrames({src, timestampsInSeconds, onFrame, signal}: ExtractFramesProps): Promise<void> {
  const input = new Input({
    formats: ALL_FORMATS,
    source: new UrlSource(src),
  });
 
  const dispose = () => {
    input.dispose();
  };
 
  if (signal) {
    signal.addEventListener('abort', dispose, {once: true});
  }
 
  try {
    const [durationInSeconds, format, videoTrack] = await Promise.all([input.computeDuration(), input.getFormat(), input.getPrimaryVideoTrack()]);
    if (!videoTrack) {
      throw new Error('No video track found in the input');
    }
 
    const timestamps =
      typeof timestampsInSeconds === 'function'
        ? await timestampsInSeconds({
            track: {
              width: videoTrack.displayWidth,
              height: videoTrack.displayHeight,
            },
            container: format.name,
            durationInSeconds,
          })
        : timestampsInSeconds;
 
    if (timestamps.length === 0) {
      return;
    }
 
    const sink = new VideoSampleSink(videoTrack);
 
    for await (const videoSample of sink.samplesAtTimestamps(timestamps)) {
      if (signal?.aborted) {
        break;
      }
 
      if (!videoSample) {
        continue;
      }
 
      const videoFrame = videoSample.toVideoFrame();
 
      onFrame(videoFrame);
    }
  } catch (error) {
    if (error instanceof InputDisposedError) {
      return;
    }
 
    throw error;
  } finally {
    dispose();
    if (signal) {
      signal.removeEventListener('abort', dispose);
    }
  }
}

Usage​

Basic example: Extract frames at specific times​

ts
await extractFrames({
  src: 'https://remotion.media/video.mp4',
  timestampsInSeconds: [0, 1, 2, 3, 4],
  onFrame: (frame) => {
    // Draw frame to canvas
    const canvas = document.createElement('canvas');
    canvas.width = frame.displayWidth;
    canvas.height = frame.displayHeight;
    const ctx = canvas.getContext('2d');
    ctx.drawImage(frame, 0, 0);
    // Don't forget to close the frame when done
    frame.close();
  },
});

Advanced: Create a filmstrip​

Extract as many frames as fit in a canvas based on the video's aspect ratio:

ts
const canvasWidth = 500;
const canvasHeight = 80;
const fromSeconds = 0;
const toSeconds = 10;
await extractFrames({
  src: 'https://example.com/video.mp4',
  timestampsInSeconds: async ({track, durationInSeconds}) => {
    const aspectRatio = track.width / track.height;
    const amountOfFramesFit = Math.ceil(canvasWidth / (canvasHeight * aspectRatio));
    const segmentDuration = toSeconds - fromSeconds;
    const timestamps: number[] = [];
    for (let i = 0; i < amountOfFramesFit; i++) {
      timestamps.push(fromSeconds + (segmentDuration / amountOfFramesFit) * (i + 0.5));
    }
    return timestamps;
  },
  onFrame: (frame) => {
    // Process each frame
    console.log(`Frame at ${frame.timestamp / 1_000_000}s`);
    frame.close();
  },
});

Important notes​

Memory management​

Always close VideoFrame objects when you're done with them to prevent memory leaks:

ts
onFrame: (frame) => {
  // Use the frame
  ctx.drawImage(frame, 0, 0);
  // Clean up
  frame.close();
};

Abort handling​

Use an AbortSignal to cancel frame extraction:

ts
const controller = new AbortController();
// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000);
await extractFrames({
  src: 'https://example.com/video.mp4',
  timestampsInSeconds: [0, 1, 2, 3, 4],
  onFrame: (frame) => {
    // Process frame
    frame.close();
  },
  signal: controller.signal,
});

VideoFrame timestamps​

VideoFrame.timestamp is in microseconds, while mediabunny's VideoSample.timestamp is in seconds. The conversion is handled automatically by .toVideoFrame().

See also​