How to Use MuseTalk with a Single Photo (Image + Audio)?

by

Making a video talk is one thing. Taking a single still photograph and having it speak is where MuseTalk starts feeling genuinely useful for people who don’t have footage to work with.

Portrait photos, product spokespersons, historical figures, character art, avatars for a project. If you have one good image and an audio file, you have enough to work with. This tutorial covers three ways to get there, and explains which one suits which situation.

First, Understand What You’re Actually Getting

Setting expectations here saves disappointment later, so let’s be direct about it.

MuseTalk changes the mouth region and nothing else. Fed a single photo, it will animate the mouth beautifully while the head stays perfectly still, the eyes never blink, and the expression never shifts. What you get is a photograph with a moving mouth.

For some uses that’s completely fine. A tight portrait delivering a short line reads well enough. For anything longer, the stillness starts to feel unnatural, because real people move constantly even when they’re standing still.

That’s why the third method below exists, and why it produces noticeably better results for longer clips. Read all three before picking one.

Method 1: Point MuseTalk Directly at Your Image

The simplest route. MuseTalk’s inference accepts an image path in place of a video path, so you can feed a photo straight through the normal pipeline.

Set up your config file the same way you would for video, but point video_path at your image:

task_0:
  video_path: "data/image/portrait.png"
  audio_path: "data/audio/speech.wav"

Then run the standard inference command for your platform.

Linux:

sh inference.sh v1.5 normal

Windows:

python -m scripts.inference --inference_config configs\inference\test.yaml --result_dir results\test --unet_model_path models\musetalkV15\unet.pth --unet_config models\musetalkV15\musetalk.json --version v15 --ffmpeg_path ffmpeg-master-latest-win64-gpl-shared\bin

Remember to point --ffmpeg_path at your own installation rather than leaving the example string in place.

The model holds your single frame steady and generates mouth movement across it for the length of your audio. Quick, straightforward, and enough for short clips.

Method 2: Convert Your Photo to Video First

If Method 1 gives you trouble, or you want more control over frame rate and duration, converting the photo into a static video with FFmpeg first is a reliable fallback.

ffmpeg -loop 1 -i portrait.png -c:v libx264 -t 10 -pix_fmt yuv420p -r 25 output.mp4

Breaking that down:

  • -loop 1 repeats the single image
  • -t 10 sets the length in seconds. Match this to your audio length or slightly exceed it
  • -r 25 sets 25 frames per second, which is what MuseTalk was trained on
  • -pix_fmt yuv420p keeps the output broadly compatible

To match your audio length automatically rather than guessing, check the duration first:

ffmpeg -i speech.wav

Then use that number for -t.

From here, treat output.mp4 as a normal source video and follow the standard lip sync process. The advantage of this route is that you control frame rate and duration explicitly, which removes two variables from any troubleshooting later.

Method 3: Animate the Photo First, Then Lip Sync (Best Results)

This is the approach worth knowing about, and it’s what the MuseTalk team designed the model to sit alongside.

MuseV is a video generation project from the same Lyra Lab team. It takes a still image and generates motion from it, producing a video where the person moves naturally. Run that output through MuseTalk afterward and you get a clip where the head moves, the expression shifts, and the mouth matches your audio.

The pipeline looks like this:

  1. Feed your photo into MuseV to generate a video with natural movement
  2. Take that generated video as your source
  3. Run MuseTalk against it with your audio file
  4. The result has both movement and accurate lip sync

The difference is substantial for anything beyond a few seconds. A head that shifts slightly and blinks occasionally stops reading as a photograph with a moving mouth and starts reading as a person talking.

If you’d rather do this without command line work, the community ComfyUI node supports both MuseV and MuseTalk, so you can wire the whole pipeline together as a node graph on one canvas. That’s covered in the ComfyUI setup guide.

Choosing Your Source Photo

This matters more than any setting you’ll adjust afterward. MuseTalk generates the mouth region at 256×256 pixels, so how your face sits in the frame directly determines how sharp the result looks.

What works well:

  • A face filling a good portion of the frame rather than sitting small in a wide shot
  • Front facing, or close to it
  • Mouth fully visible and unobstructed
  • Even lighting across the lower half of the face
  • Neutral or slightly open mouth position
  • Sharp focus, particularly around the mouth

What causes trouble:

  • Faces small in a large image, which get upscaled from very little detail
  • Strong profile angles, where face detection often fails outright
  • Hands, microphones, or hair crossing the jaw
  • Heavy shadow across the mouth
  • A broad open smile in the source, which constrains what the model can generate from it
  • Facial hair around the mouth, since the model doesn’t fully preserve details like moustaches

A practical tip: if your first result looks soft, crop the image tighter around the face and rerun before adjusting anything else. This single change fixes more quality complaints than any parameter does.

Getting the Mouth Movement Right

Once you have output, bbox_shift is the setting that will improve it most.

It controls where the mask boundary sits, which changes how open the generated mouth appears. Positive values increase openness, negative values reduce it, and the default is 0.

The workflow that works:

  1. Run once at the default
  2. Note the suggested adjustable range reported back for your input
  3. Run again with a value inside that range

Single photos have a particular tendency worth knowing about. Because your source is one fixed expression rather than a face already moving through a range of shapes, the model has less variation to draw on.

If your photo has a closed or neutral mouth, a positive bbox_shift value often helps the generated speech look more natural.

If your photo already has an open mouth, you’ll usually want less adjustment, not more.

Quick Fixes for Common Problems

No face detected

The face is likely too small, too angled, or too poorly lit. Crop tighter, pick a more front facing image, or try a photo with clearer lighting on the lower face.

The result looks like a photo with a moving mouth

That’s exactly what Method 1 and 2 produce. If it bothers you, Method 3 is the answer. There’s no setting that adds head movement, because MuseTalk doesn’t generate head movement at all.

The mouth region looks blurry against a sharp face

Your face is too small in frame for the 256×256 generation region to hold up. Crop tighter and rerun. Turning off the fp16 flag helps, and a face restoration pass afterward can recover some detail.

A visible edge or seam around the mouth

Adjust bbox_shift and rerun, since it moves where that boundary sits. Facial hair around the mouth makes this harder to hide, which is a known limitation rather than a configuration mistake.

The mouth barely moves

Push bbox_shift positive, and confirm your audio contains clear speech rather than something quiet or heavily mixed with music.

Output duration doesn’t match the audio

More likely with Method 2, where you set the video length manually. Make sure your -t value in the FFmpeg command matches or slightly exceeds your audio length.

Leave a Comment