MuseTalk uses speech audio to synthesize the lower face in a source image or video. It combines audio features with a compact representation of a face crop, decodes the result and blends it into the source frame. A single-photo input remains a static-head workflow; head movement in a source video comes from that video.
The pipeline in four steps
| Stage | What happens | What it does not guarantee |
|---|---|---|
| 1. Prepare the face | Locate the face, crop and resize it to 256 × 256, encode masked/reference information with a VAE, and prepare blending masks. | The output canvas size does not restore fine detail lost in the face crop. |
| 2. Process speech | Whisper supplies audio features aligned with the target frame rate. | This is not a promise of accurate lip sync for every voice, language or recording. |
| 3. Synthesize the face crop | An audio-conditioned U-Net predicts the face latent in one forward pass; the VAE decoder converts it to pixels. | One network pass does not make the whole application instant. |
| 4. Blend and deliver | Resize the generated crop, blend it into the source frame, then optionally save frames, encode video and combine audio. | Blending can change cheeks and skin detail around the mouth. Encoding and delivery add time. |
The model uses components associated with Stable Diffusion, but inference does not run a multi-step diffusion denoising schedule. The official project describes this as audio-driven latent-space inpainting.
What a prepared avatar saves
The real-time example can cache source frames, face coordinates, face latents and blending masks. Reusing that avatar avoids preparing the same visual material for every audio request. A fresh process still loads models and reads cached avatar data; keeping the process alive can avoid repeating those startup costs.
New speech still requires audio processing. Frame synthesis, decoding and blending still run, and a saved MP4 also needs encoding and audio muxing. The pinned real-time script takes complete audio files and simulates repeated requests; it does not itself demonstrate microphone-to-browser streaming.
Throughput and latency answer different questions
Throughput describes how many frames a measured stage processes per second. End-to-end latency describes how long the user waits for a usable result. Model startup, avatar preparation or loading, audio processing, frame generation, encoding and network transfer belong in the relevant latency total. A generation-only FPS figure cannot stand in for that total.
See the measured real-time timing guide for hardware, stage boundaries and cold/prepared-avatar results.
Capabilities and limits in practice
- Use an existing video when its head movement and framing should be retained. MuseTalk is not a general body-motion generator.
- With a single photo, expect a static base image. Natural head turns and blinks require a separate motion pipeline, which adds its own work and limitations.
- Only use controls supported by the selected version. In the tested code, 1.0 uses bbox_shift for crop adjustment; 1.5 forces bbox_shift to zero and supports extra_margin and configurable blending modes.
- Judge complete clips with sound and inspect both the whole frame and mouth crop. Static samples cannot establish temporal stability or accurate audio alignment.
Start with the single-photo tutorial, the video lip-sync tutorial, or the quality guide with real matched examples.