Skip to main content

Using a video as a texture in Three.js

If you want to embed a video as a texture in Three.js, you have multiple options.

Using @remotion/mediav4.0.387​

We recommend that you mount a <Video> tag in headless mode and use the onVideoFrame prop to update the Three.js texture whenever a new frame is being drawn.

VideoTexture.tsx
tsx
import {useThree} from '@react-three/fiber';
import {Video} from '@remotion/media';
import {ThreeCanvas} from '@remotion/three';
import React, {useCallback, useState} from 'react';
import {useVideoConfig} from 'remotion';
import {CanvasTexture} from 'three';
 
const videoSrc = 'https://remotion.media/video.mp4';
 
const videoWidth = 1920;
const videoHeight = 1080;
const aspectRatio = videoWidth / videoHeight;
const scale = 3;
const planeHeight = scale;
const planeWidth = aspectRatio * scale;
 
const Inner: React.FC = () => {
  const [canvasStuff] = useState(() => {
    const canvas = new OffscreenCanvas(videoWidth, videoHeight);
    const context = canvas.getContext('2d')!;
    const texture = new CanvasTexture(canvas);
    return {canvas, context, texture};
  });
 
  const {invalidate} = useThree();
 
  const onVideoFrame = useCallback(
    (frame: CanvasImageSource) => {
      canvasStuff.context.drawImage(frame, 0, 0, videoWidth, videoHeight);
      // This is needed during preview - the frame loop synchronizes with the monitor refresh rate
      // By setting this, we signal that the texture has been updated
      canvasStuff.texture.needsUpdate = true;
      // This is needed during rendering - the frame loop is only triggered on demand.
      // We have to call "invalidate()" to trigger a new frame loop.
      invalidate();
    },
    [canvasStuff.context, canvasStuff.texture, invalidate],
  );
 
  return (
    <>
      <Video src={videoSrc} onVideoFrame={onVideoFrame} muted headless />
      <mesh>
        <planeGeometry args={[planeWidth, planeHeight]} />
        <meshBasicMaterial color={0xffffff} toneMapped={false} map={canvasStuff.texture} />
      </mesh>
    </>
  );
};
 
export const RemotionMediaVideoTexture: React.FC = () => {
  const {width, height} = useVideoConfig();
 
  return (
    <ThreeCanvas style={{backgroundColor: 'white'}} linear width={width} height={height}>
      <Inner />
    </ThreeCanvas>
  );
};

Notes​

  • By using the headless prop, nothing will be returned by the <Video> tag, so it can be mounted within a <ThreeCanvas> without affecting the rendering.
  • For preview and rendering, separate approaches are needed to invalida the texture when it is being updated. See

Examples​

Using <OffthreadVideo>​

deprecated in favor of using @remotion/media

You can use the useOffthreadVideoTexture() hook from @remotion/three to get a texture from a video.

Drawbacks:

  • It requires the whole video to be downloaded to disk first before frames can be extracted
  • It does not work in client-side rendering
  • It creates a new texture for each frame, which is less efficient than using @remotion/media

This API is therefore deprecated in favor of using the recommended approach mentioned above.

Using <Html5Video>​

deprecated in favor of using @remotion/media

You can use the useVideoTexture() hook from @remotion/three to get a texture from a video.

It has all the drawbacks of the <Html5Video> tag and is therefore deprecated in favor of using the recommended approach mentioned above.

See also​