You’ve got MuseTalk installed, the weights are sitting in the right folders, and FFmpeg finally stopped complaining. Now comes the part you actually wanted: taking a video of someone talking and making their mouth match a completely different audio track.
This tutorial covers the full process from preparing your files through to a finished clip, including the settings that separate a result that looks convincing from one that looks obviously generated. We’ll work through the standard command line method, since that’s where the real control lives.
What You’ll Need Before Starting
Confirm all of this is ready:
- A working MuseTalk installation with the conda environment activated
- FFmpeg installed and reachable from your terminal
- Model weights downloaded and organised in the correct folder structure
- A source video containing a clear face
- An audio file with the speech you want the mouth to match
If any of those are shaky, sort them out first. Almost every problem people hit during their first lip sync run traces back to setup rather than the process itself.
How to Lip Sync a Video Using MuseTalk?
Step 1: Prepare Your Source Video
This step gets skipped constantly, and it’s responsible for a large share of disappointing results. Ten minutes here saves an hour of confused rerunning later.
Match the frame rate to 25fps
MuseTalk was trained on video running at 25 frames per second. Feed it something else and the timing can drift, sometimes subtly enough that you can’t quite say what looks wrong.
Check what you’re working with:
ffmpeg -i yourvideo.mp4
Look for the fps value in the output. If it isn’t 25, convert it:
ffmpeg -i input.mp4 -r 25 output.mp4
If your source runs at a noticeably lower frame rate, straight conversion duplicates frames rather than creating new ones. Frame interpolation gives a better result in that situation, though for most footage a simple conversion is fine.
Check what’s actually in frame
MuseTalk generates the mouth region at 256×256 pixels. That number tells you everything about what makes a good source.
A face filling a decent portion of the frame gives the model plenty of detail to work with.
A face sitting small in a wide shot gets upscaled from very little information, and the result looks soft no matter what settings you use.
If your subject is far from the camera, crop tighter before processing.
Beyond that, look for:
- An unobstructed mouth. Hands near the face, microphones in shot, and hair across the jaw all cause trouble.
- Reasonably steady framing. Rapid head movement and motion blur make the model’s job harder.
- A mostly front facing angle. Extreme profile shots are where face detection tends to fail outright.
Trim to something short for testing
Work with a ten second clip while you’re finding your settings. Once you know what works, run the full length version. Waiting five minutes to discover the mouth openness is wrong, then waiting five minutes again, gets old fast.
Step 2: Prepare Your Audio
Clean speech produces clean lip movement. There’s not much more to it than that, but the difference is real.
What helps:
- Speech isolated from music and background noise
- One voice rather than overlapping speakers
- Minimal room echo
- Consistent volume without long silent gaps
MuseTalk reads the shape of the sound rather than the words, so anything muddying that shape works against you.
If you have access to a clean vocal recording rather than a final mix, use it.
Step 3: Set Up Your Config File
MuseTalk reads your input paths from a YAML config file rather than taking them as command arguments. The repository includes an example at configs/inference/test.yaml that you can edit directly or copy.
The structure is straightforward:
task_0:
video_path: "data/video/yourvideo.mp4"
audio_path: "data/audio/youraudio.wav"
Point those two paths at your own files. You can add more tasks in the same file if you want to process several clips in one run:
task_0:
video_path: "data/video/clip_one.mp4"
audio_path: "data/audio/audio_one.wav"
task_1:
video_path: "data/video/clip_two.mp4"
audio_path: "data/audio/audio_two.wav"
A practical note: relative paths are read from your MuseTalk project folder, so run your commands from there. If paths keep failing, switching to absolute paths removes the ambiguity entirely.
Step 4: Run Your First Generation
Now the actual work. Pick the command for your platform.
Linux
sh inference.sh v1.5 normal
Swap v1.5 for v1.0 if you want the older model version.
Windows
python -m scripts.inference --inference_config configs\inference\test.yaml --result_dir results\test --unet_model_path models\musetalkV15\unet.pth --unet_config models\musetalkV15\musetalk.json --version v15 --ffmpeg_path ffmpeg-master-latest-win64-gpl-shared\bin
Two arguments need your attention here. The --ffmpeg_path value has to match your own FFmpeg installation, not the example string. And if you’d rather run version 1.0, point --unet_model_path at models\musetalk\pytorch_model.bin, --unet_config at models\musetalk\musetalk.json, and change --version to v1.
Your finished video lands in the results directory you specified.
Step 5: Adjust bbox_shift (The Setting That Matters Most)
Watch your first result. If the mouth moves but something feels off, this is almost certainly why.
bbox_shift controls where the mask boundary sits on the face, which changes how open the generated mouth appears.
- Positive values move the boundary downward and increase mouth openness
- Negative values move it upward and reduce mouth openness
- The default is 0
The approach the MuseTalk team recommends works well in practice:
- Run once at the default value
- The output reports back a suggested adjustable range calculated from your specific input
- Run again with a value inside that range, pushing toward more or less openness based on what you saw
Which direction do you need?
A useful rule of thumb: if your source video shows a face with the mouth mostly closed and fairly still, a positive value usually helps, since the model needs encouragement to open the mouth convincingly. If your source already shows an animated, expressive speaker, a smaller or negative adjustment often looks more natural.
There’s no universally correct number. It depends on the face, the framing, and the audio. That second pass is where most of the quality improvement in a MuseTalk workflow actually comes from, so it’s worth the extra render.
Step 6: Refine Your Final Render
Once your settings are dialled in on the test clip, a few final adjustments before the full length run.
Drop the fp16 flag if your GPU allows.
Running without --use_float16 produces better quality output at the cost of more VRAM and longer generation time. Use fp16 while testing, full precision for the version you’re keeping.
Consider upscaling.
MuseTalk’s 256×256 mouth region is its known resolution ceiling. Running the finished video through a face restoration tool like GFPGAN afterward can sharpen the result noticeably, though it adds a processing step and can subtly alter appearance.
Check the whole clip, not just the start.
Lip sync issues sometimes appear midway through, particularly around pauses or sudden changes in speaking pace.
Real Time Generation
If you want output as audio arrives rather than rendering a file afterward, MuseTalk includes a separate real time pipeline.
Linux
sh inference.sh v1.5 realtime
Windows
python -m scripts.realtime_inference --inference_config configs\inference\realtime.yaml --result_dir results\realtime --unet_model_path models\musetalkV15\unet.pth --unet_config models\musetalkV15\musetalk.json --version v15 --fps 25 --ffmpeg_path ffmpeg-master-latest-win64-gpl-shared\bin
The real time setup uses a different config file, realtime.yaml, which works on an avatar concept. You prepare an avatar once from a source video, and after that you can feed it different audio clips without redoing the preparation each time. The config includes a preparation flag that you set to true the first time and false on subsequent runs against the same avatar.
Fair warning on hardware: the 30fps and above figure comes from an NVIDIA Tesla V100. Real time performance on consumer cards varies considerably, and on smaller GPUs “real time” becomes aspirational.
Common Problems and Fixes
No face detected, or the run fails immediately
Check that a face is clearly visible and reasonably front facing throughout your clip. A single frame where the face turns away or gets obscured can stop the whole run.
The mouth barely moves
Increase bbox_shift toward a positive value. Also confirm your audio contains clear, audible speech rather than something very quiet or heavily mixed with music.
Mouth movement looks exaggerated or unnatural
Move bbox_shift in the negative direction and rerun.
Timing drifts out of sync partway through
Frame rate mismatch is the usual cause. Convert your source to 25fps and try again.
The mouth region looks blurry against a sharp face
Your face is likely too small in frame. Crop tighter and rerun. Turning off fp16 helps too, and a face restoration pass afterward can recover more detail.
Visible edges or a seam around the mouth
Adjusting bbox_shift usually improves this, since it moves where the boundary sits. Faces with facial hair around the mouth are harder here, since identity details like a moustache aren’t fully preserved. That’s a known limitation of the model rather than something you’ve configured wrong.
Slight jitter between frames
MuseTalk generates each frame individually, so a small amount of frame to frame variation is expected. The team lists this as a known limitation. Version 1.5 handles it better than 1.0 if you’re still on the older weights.
Out of memory
Add --use_float16, shorten your clip, reduce resolution, and close other GPU heavy applications.
Setting Realistic Expectations
MuseTalk is genuinely good at what it does, and it’s worth knowing where its edges are so you’re not chasing a result it can’t produce.
The model changes the mouth region and nothing else. Head position, eye movement, and expression all stay exactly as they were in your source. If you pair a calm, still speaker with energetic audio, the mismatch shows, because only the mouth reacts.
The team is also open about three specific limitations: the face resolution has room to improve, identity details like moustaches and precise lip shape aren’t fully preserved, and single frame generation introduces some jitter.
Knowing this shapes how you choose source material. A source video where the speaker’s natural energy roughly matches your target audio will always outperform a mismatched pairing, no matter how carefully you tune your settings.