Skip to main content

Using @remotion/video-matting in Node.jsv4.0.528

You can use @remotion/video-matting in Node.js to separate a local video into an opaque base and a transparent foreground.

Register @mediabunny/server to enable video decoding, VP9 encoding with alpha, and audio processing. A browser or canvas polyfill is not required.

Installation​

npm i --save-exact @remotion/[email protected] @huggingface/transformers mediabunny @mediabunny/server
This assumes you are currently using v4.0.527 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

Example​

separate-video.ts
import {registerMediabunnyServer} from '@mediabunny/server';
import {
  canUseVideoMatting,
  downloadVideoMattingModel,
  loadVideoMattingModel,
  separateVideoLayers,
} from '@remotion/video-matting';
import {writeFile} from 'node:fs/promises';

registerMediabunnyServer();

const model = 'modnet';
const support = await canUseVideoMatting({model});
if (!support.supported) {
  throw new Error(support.detailedReason);
}

await downloadVideoMattingModel({model});

// In long-running apps, put this code in an async function so resources
// are released when that function finishes.
await using modelHandle = await loadVideoMattingModel({model});
await using layers = await separateVideoLayers({
  src: './input.mp4',
  model,
  audio: 'base',
  onProgress: ({processedFrames}) => console.log(processedFrames),
});

await writeFile(
  'base.webm',
  new Uint8Array(await (await layers.base.getBlob()).arrayBuffer()),
);
await writeFile(
  'foreground.webm',
  new Uint8Array(await (await layers.foreground.getBlob()).arrayBuffer()),
);

separateVideoLayers() accepts local file paths, file: URLs, remote URLs, and Blob inputs. Outputs are held in memory by default. Use outputWritable to stream large outputs; handle each chunk's position because writes are not necessarily append-only.

Model cache​

Model files use the Transformers.js filesystem cache. To choose the directory, set env.cacheDir before downloading or loading models:

model-cache.ts
import {env} from '@huggingface/transformers';
import {resolve} from 'node:path';

env.cacheDir = resolve('.cache/video-matting');

removeVideoMattingModel() removes cached files. Disposing a loaded model releases its memory and keeps the downloaded files.

Requirements​

A compatible GPU is required. Transformers.js uses ONNX Runtime's native WebGPU provider in Node.js. canUseVideoMatting() runs a tiny inference to check support, including fp16 computation for ben2-base.

The native ONNX Runtime dependency does not support Linux arm64. GPU availability also depends on the host and its drivers; a typical serverless environment does not provide a GPU.

See also​