live2d-web documentation

MediaPipe face tracking

Attach standard or Perfect Sync face tracking on main or Worker.

Main-thread tracking

MediaPipe is an optional peer and loads only from its subpath. You supply WASM, the Face Landmarker model, video frames and scheduling. The tracker keeps normalized values, never frames or landmarks.

import { createMediaPipeFaceTracker } from
  'live2d-web/tracking/mediapipe'
 
const tracker = await createMediaPipeFaceTracker({
  modelAssetPath: '/mediapipe/face_landmarker.task',
  wasmPath: '/mediapipe/wasm',
})
const detach = tracker.attach(character, {
  mapping: 'auto',
  channels: { mouth: false },
})
 
tracker.update(video, performance.now())
 
detach()
tracker.dispose()

Optional Worker

Worker mode keeps inference away from rendering. update() becomes asynchronous, accepts one frame at a time and reports busy frames as skipped. If Worker startup fails, choose main mode explicitly; there is no silent fallback. Initialization has a 30-second deadline and inference has a 10-second deadline. A timeout rejects with tracking-error, terminates that tracker and requires the app to create a new Worker tracker or switch to main mode.

const tracker = await createMediaPipeFaceTracker({
  execution: 'worker',
  modelAssetPath: '/mediapipe/face_landmarker.task',
  wasmPath: '/mediapipe/wasm',
  workerFactory: () => new Worker(
    new URL('./face-tracking.worker.ts', import.meta.url),
    { type: 'module' },
  ),
})

Startup and mobile status

The Playground separates camera readiness, tracker creation, first inference, one-second calibration and the final tracked transition. It also lists matching MediaPipe JavaScript, WASM and task Resource Timing entries. Use those phases before treating a long cold start as inference work.

Warm local tracker creation is measured against a five-second median and is currently well below that budget on the three desktop test engines. That five-second number is a local expectation checked by hand, not a CI gate: no workflow runs the tracking benchmark. iPhone Safari still requires physical-device validation, and Android Chrome is explicitly unverified. If Worker creation fails or times out, surface the tracking-error and let the user recreate the tracker in main mode; do not wait indefinitely or silently fall back.

Calibrate and map deliberately

Keep a neutral face in view during the initial one-second calibration. Call calibrate() again after camera position or lighting changes. auto uses Perfect Sync only when enough matching parameters exist; otherwise it selects the standard pose, eyes, brows, mouth and cheek channels. Disable mouth when a microphone owns ParamMouthOpenY.

Own capture and teardown in the app

The tracker does not request camera permission or schedule frames. The app owns the <video>, MediaStream, requestAnimationFrame and visibility lifecycle. On stop, detach the target, dispose the tracker, cancel frame production and stop camera tracks. A Worker tracker also closes transferred ImageBitmap objects and terminates its Worker during dispose.