Real-Time AI Lip-Sync
MuseTalk is an open-source AI model that matches a person’s lip movements to any audio track, in real time, at over 30 frames per second. Built by Tencent Music’s Lyra Lab, it’s one of the fastest and most accurate open-source lip-sync tools available today.
This site is dedicated entirely to MuseTalk — how it works, how to install it, how to use it, and how it compares to other tools. If you just heard about MuseTalk for the first time, or you’re a developer setting up a training pipeline, you’ll find a clear answer here.
What is MuseTalk?
MuseTalk is a real-time, high-quality lip-syncing model. In plain terms: you give it a face (a video or a single photo) and an audio clip, and it generates output where the mouth moves to match the audio — even in a language the original speaker never said.
A few things make it stand out from most lip-sync tools you’ll come across:
- It’s fast. MuseTalk runs at 30fps+ on an NVIDIA Tesla V100, which is fast enough for live, real-time use rather than just offline rendering.
- It’s multilingual. The model supports audio in Chinese, English, and Japanese out of the box.
- It works with photos, not just video. You can animate a single still image using nothing but an audio file.
- It’s genuinely open-source. The code, trained weights, and even the training scripts are all public, and the license allows commercial use.
- It pairs with MuseV. For a full virtual-human pipeline, you can generate a video with MuseV first, then use MuseTalk to add accurate lip movement on top of it.
MuseTalk isn’t trying to generate a whole face or body from scratch — it focuses specifically on the mouth and lower-face region, modifying a 256×256 area around the mouth so the rest of the identity, lighting, and background stay untouched.
Common uses people are building with it right now: dubbing videos into other languages, giving a static photo a talking presence, building AI avatars and virtual hosts, and adding speech to game or e-learning characters.
How Does MuseTalk Work?
Most people assume MuseTalk is a diffusion model, since it borrows parts of its design from Stable Diffusion. It isn’t. That distinction actually explains why it’s so fast.
Here’s the simplified version of what happens behind the scenes:
- The face gets compressed. A frozen VAE (variational autoencoder) takes the video frame and shrinks it down into a compact numerical representation called a latent — think of it as a highly compressed sketch of the image that a neural network can work with efficiently.
- The lower half of the face is masked out. MuseTalk covers the mouth and jaw region of that latent, along with a reference image of the same face, so the model has context for what the person looks like without knowing what the mouth should be doing.
- Audio gets turned into meaning. A frozen Whisper-tiny model converts the audio into an embedding — a numerical representation of the sounds being spoken.
- A U-Net fills in the gap. The audio embedding and the face’s latent are merged using cross-attention inside a U-Net (borrowed in structure from Stable Diffusion), which fills in — or “inpaints” — the missing mouth region so it matches the audio.
- The result gets decoded back into pixels. The VAE decoder turns the finished latent back into a normal video frame.
Because this entire process happens in one pass, rather than through the many repeated denoising steps a diffusion model needs, MuseTalk can generate frames fast enough for live, real-time output.
MuseTalk 1.0 vs MuseTalk 1.5: What’s New and Different
MuseTalk has gone through a meaningful upgrade since its original release, and the differences matter if you’re deciding which version to run.
| MuseTalk 1.0 | MuseTalk 1.5 | |
|---|---|---|
| Released | April 2024 | March 2025 |
| Training approach | Primarily L1 loss | Adds perceptual loss, GAN loss, and sync loss |
| Training strategy | Single-stage | Two-stage training with spatio-temporal data sampling |
| Lip-sync accuracy | Good | Noticeably improved |
| Identity consistency | Moderate | Improved |
| Clarity/sharpness | Moderate | Improved |
| Mouth control | bbox_shift parameter | Same parameter, refined results |
The core idea behind the 1.5 upgrade was striking a better balance between two things that usually trade off against each other: how good a frame looks on its own, and how accurately the lips track the audio. Adding GAN and perceptual loss pushed image quality up, while the sync loss and the new data sampling approach pushed lip accuracy up at the same time, rather than sacrificing one for the other.
As of the most recent update, the training code for MuseTalk is fully public too, so anyone can train or fine-tune their own version using the same two-stage process.
Recommendation: unless you have a specific reason to use 1.0 (older tutorials, existing pipeline compatibility), MuseTalk 1.5 is the version to start with.
Is MuseTalk Free to Use?
Yes — and this is one of the more generous licenses you’ll find among open-source AI face/video tools.
The MuseTalk code is released under the MIT License, with no restriction on academic or commercial use.
The trained model weights are also available for any purpose, including commercial projects.
Third-party components MuseTalk depends on — Whisper, the VAE, DWPose, and similar — each carry their own license, so if you’re building a commercial product, it’s worth checking those individually rather than assuming MuseTalk’s license alone covers everything.
Sample/test data included in the repository is collected from the internet and is for non-commercial research only — that restriction applies to the demo data, not to videos you generate yourself with your own inputs.
Responsible use is on you. The MuseTalk team is explicit that while people are free to create content with the tool, users are expected to follow local laws, and the developers don’t take responsibility for misuse.
In short: you can build a commercial product on top of MuseTalk, but double-check the licenses of the supporting models it relies on, and use it responsibly given how sensitive face and voice manipulation tools can be.
MuseTalk Architecture Explained (VAE + Whisper + UNet, In Plain English)
MuseTalk is really three separate, well-established AI components wired together in a clever way, rather than one giant new invention. Here’s what each piece is doing:
1. The VAE (ft-mse-vae) — a frozen, pre-trained image compressor borrowed from the Stable Diffusion ecosystem. It doesn’t get retrained here; its only job is turning video frames into compact latents and turning finished latents back into full images.
2. Whisper-tiny — OpenAI’s small speech-recognition model, also frozen. Instead of using it to transcribe words, MuseTalk uses its internal audio embeddings — the numerical patterns it learned about sound — as a rich description of what’s being spoken.
3. The U-Net — the only part that’s actually trained specifically for MuseTalk. Structurally, it’s based on the U-Net from Stable Diffusion v1.4, but its job here is different: instead of denoising an image step-by-step, it takes the masked face latent and the audio embedding and fuses them together through cross-attention, in a single step, to fill in the correct mouth shape.
Put together: frozen VAE + frozen Whisper + trained U-Net = a system that can inpaint an accurate, audio-matched mouth onto any face, in a single fast pass. That single-step design (instead of the multi-step process diffusion models use) is the main reason MuseTalk is fast enough for real-time output.
Who Made MuseTalk? Tencent Music’s Lyra Lab
MuseTalk comes out of Lyra Lab, a research team at Tencent Music Entertainment.
The team has been actively maintaining the project since its first release in April 2024, releasing the real-time inference pipeline, a public Gradio demo on Hugging Face, a full technical report, the 1.5 model upgrade, and, as of April 2025, the complete training code.
MuseTalk is also designed to work alongside MuseV, a related video-generation project from the same lab, as part of a broader virtual-human toolkit.
The project is hosted publicly on GitHub, with a live demo on Hugging Face Spaces and pretrained weights on the Hugging Face Hub, and it has attracted a large open-source community — including third-party integrations like a dedicated ComfyUI node.
MuseTalk Technical Report Summary (For Non-Researchers)
MuseTalk’s official technical report, titled “MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling,” is published on arXiv. Here’s what it actually says, without the academic language:
The problem they were solving: getting a dubbed face to look right (high resolution, consistent identity) and sound-matched (accurate lip movement) at the same time is hard — and it’s especially hard if you also need it to run fast enough for live streaming, not just pre-rendered video.
Their approach: rather than using a diffusion model — which needs many repeated steps to generate an image — the team designed a system that encodes the face into a latent space and fills in the mouth region in a single step. This is what makes real-time generation possible.
Their training innovation: they introduce a sampling strategy that specifically picks reference frames where the head pose closely matches the target frame during training. This forces the model to focus its attention on getting the mouth movement right, instead of “wasting” its learning capacity on figuring out head position or angle differences that shouldn’t affect lip shape anyway.
Their finding on training signals: the paper also digs into how the lip-sync loss function relates to how much audio information the model is given — essentially explaining why certain training choices lead to better or worse mouth accuracy.
The result they report: MuseTalk consistently outperformed recent state-of-the-art methods on image quality, while matching them on lip-sync accuracy — and it can generate 256×256 faces at 30fps+ with almost no startup delay, which is what unlocks real-time applications like live avatars and streaming.