MuseTalk Installation: Step by Step for Windows and Linux

Setting up MuseTalk takes some preparation because it depends on Python, PyTorch, CUDA, FFmpeg, MMLab packages, and several model weights. This guide combines the standard MuseTalk installation process with the fixes needed for modern Vast.ai GPU instances.

The guide covers Windows and Linux, with MuseTalk 1.5 as the recommended version.

The most important rule is to keep the software versions consistent. Do not randomly upgrade PyTorch, Transformers, Hugging Face Hub, or xformers after the environment is working.

My Setup

For this setup, I use SSH (Secure Shell) to connect from my local computer to the Vast.ai GPU instance. Before connecting, add your computer’s public SSH key to your Vast.ai account under the SSH Keys section. On macOS or Linux, you can generate a key with ssh-keygen -t ed25519 and copy the contents of the .pub file to Vast.ai.

Once the key is added, open your Vast.ai instance and click Connect → SSH to get the SSH connection command. Run that command in your local Terminal to access the remote GPU machine securely without entering a password each time.

For example:

ssh -p 57488 root@47.186.21.5

Before You Begin

You should have:

  • Python 3.10
  • NVIDIA GPU
  • CUDA-compatible NVIDIA driver
  • Git
  • FFmpeg
  • Conda/Miniconda
  • A stable internet connection
  • Several GB of free disk space for model weights

For normal MuseTalk inference, a single RTX 4090 24GB is an excellent choice. You do not need multiple GPUs or an A100 for ordinary inference.

MuseTalk’s documented environment uses Python 3.10 and PyTorch 2.0.1 with CUDA 11.8 wheels.

Recommended Vast.ai configuration

For Vast.ai, use approximately:

GPU:        1 × RTX 4090 24GB
Template:   CUDA11
RAM:        16 GB minimum, 32 GB preferred
Disk:       40–50 GB

A 3090 24GB can also work, but the 4090 is preferable for speed.

Part 1: Build the Python Environment

Windows and Linux

The basic MuseTalk Python environment is the same on both platforms.

Step 1: Install Conda

If Conda is already installed, skip this section and continue to Step 2.

For a fresh Linux/Vast.ai container without Conda:

cd /root

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda

Initialize Conda:

/opt/conda/bin/conda init bash
source ~/.bashrc

Check:

conda --version

You should see something similar to:

conda 25.x.x

Important for Vast.ai

Some Vast.ai images already include Miniconda under:

/opt/conda

In that case, do not install Miniconda again.

Simply check:

conda --version

Step 2: Accept Conda Terms of Service on Vast.ai

Some recent Conda installations require accepting the Anaconda channel Terms of Service before creating an environment.

Run:

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main

Then:

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r

Step 3: Create the MuseTalk environment

Run:

conda create -n MuseTalk python=3.10 -y

Activate it:

conda activate MuseTalk

Verify:

python --version

Expected:

Python 3.10.x

Also verify:

which python

You should see something similar to:

/opt/conda/envs/MuseTalk/bin/python

Do not use Python 3.14

Some Vast.ai templates start with Python 3.14 in the base environment. MuseTalk should be run inside the dedicated Python 3.10 MuseTalk environment.

Part 2: Install PyTorch

Step 4: Install the exact PyTorch version

Do not install the newest PyTorch version.

MuseTalk’s documented combination is:

PyTorch      2.0.1
Torchvision  0.15.2
Torchaudio   2.0.2
CUDA wheel   11.8

Using pip:

python -m pip install \
  torch==2.0.1 \
  torchvision==0.15.2 \
  torchaudio==2.0.2 \
  --index-url https://download.pytorch.org/whl/cu118

MuseTalk documents this exact combination.

Verify PyTorch immediately

Run:

python -c "import torch; print('Torch:',torch.__version__); print('CUDA:',torch.version.cuda); print('CUDA available:',torch.cuda.is_available()); print('GPU:',torch.cuda.get_device_name(0))"

Expected:

Torch: 2.0.1+cu118
CUDA: 11.8
CUDA available: True
GPU: NVIDIA GeForce RTX 4090

Important CUDA clarification

On a Vast.ai machine, nvidia-smi may report something such as:

CUDA Version: 12.6

That is the CUDA capability supported by the installed NVIDIA driver. It does not mean you need to install PyTorch with CUDA 12.6.

The MuseTalk environment can still use the PyTorch cu118 build successfully.

Part 3: Clone MuseTalk

Step 5: Clone the repository

Linux/Vast.ai:

cd /workspace

git clone https://github.com/TMElyralab/MuseTalk.git

cd MuseTalk

If the repository already exists:

cd /workspace/MuseTalk

Part 4: Install Dependencies

Step 6: Install MuseTalk requirements

Run:

python -m pip install -r requirements.txt

MuseTalk’s requirements file contains the project-specific dependency versions.

Do not immediately run:

pip install -U torch

after this.

Part 5: Install MMLab Packages

MuseTalk uses the MMLab ecosystem for detection and pose processing. Its documented versions are:

MMCV   2.0.1
MMDet  3.1.0
MMPose 1.1.0

Step 7: Install OpenMIM

python -m pip install --no-cache-dir -U openmim

Then:

mim install mmengine

mim install "mmcv==2.0.1"

mim install "mmdet==3.1.0"

Step 8: Install MMPose with the chumpy workaround

MMPose 1.1.0 depends on the old chumpy package.

Modern pip build isolation can cause:

ModuleNotFoundError: No module named 'pip'

when building chumpy.

Avoid that problem by installing the build tools and chumpy separately.

Run:

python -m pip install "setuptools<82" wheel

Then:

python -m pip install --no-build-isolation "chumpy==0.70"

Finally:

python -m pip install "mmpose==1.1.0"

Part 6: Fix the Hugging Face Version

A newer Hugging Face Hub version can break the Transformers version required by MuseTalk.

Keep these versions:

Transformers       4.39.2
huggingface_hub    0.30.2

Run:

python -m pip install --force-reinstall \
  "transformers==4.39.2" \
  "huggingface_hub==0.30.2"

Verify:

python -c "import transformers,huggingface_hub; print('Transformers:',transformers.__version__); print('HF Hub:',huggingface_hub.__version__)"

Expected:

Transformers: 4.39.2
HF Hub: 0.30.2

Do not install modern xformers

Do not run:

pip install xformers

and do not upgrade PyTorch simply because a newer xformers package requests Torch 2.10 or newer.

MuseTalk works with the older PyTorch stack above.

Part 7: Remove Unnecessary OpenXLab Packages

Installing OpenMIM can leave opendatalab/OpenXLab-related packages in the environment.

If:

python -m pip check

reports:

opendatalab ... requires openxlab

you do not need to install OpenXLab just for MuseTalk inference.

After the MMLab packages have been installed, you can remove the unnecessary tooling:

python -m pip uninstall -y openmim opendatalab

Then:

python -m pip check

The MMLab packages themselves remain installed.

Part 8: Verify the Complete Python Environment

Before downloading models, test the complete environment.

Run:

python - <<'PY'
import torch
import mmcv
import mmengine
import mmdet
import mmpose
import cv2
import diffusers
import transformers
import librosa
import huggingface_hub

print("Torch:", torch.__version__)
print("CUDA:", torch.version.cuda)
print("CUDA available:", torch.cuda.is_available())
print("GPU:", torch.cuda.get_device_name(0))
print("MMCV:", mmcv.__version__)
print("MMEngine:", mmengine.__version__)
print("MMDet:", mmdet.__version__)
print("MMPose:", mmpose.__version__)
print("OpenCV:", cv2.__version__)
print("Diffusers:", diffusers.__version__)
print("Transformers:", transformers.__version__)
print("Librosa:", librosa.__version__)
print("HF Hub:", huggingface_hub.__version__)
PY

For the tested Vast.ai setup, the important values are:

Torch: 2.0.1+cu118
CUDA: 11.8
CUDA available: True
GPU: NVIDIA GeForce RTX 4090
MMCV: 2.0.1
MMDet: 3.1.0
MMPose: 1.1.0
Diffusers: 0.30.2
Transformers: 4.39.2
HF Hub: 0.30.2

Part 9: Set Up FFmpeg

MuseTalk requires FFmpeg.

Linux

Install:

apt-get update
apt-get install -y ffmpeg

Check:

which ffmpeg

Then:

ffmpeg -version | head -n 1

You should normally see:

/usr/bin/ffmpeg

If MuseTalk specifically asks for FFMPEG_PATH, you can set:

export FFMPEG_PATH=/usr/bin

For a static FFmpeg build, set FFMPEG_PATH to the directory containing your FFmpeg executable.

Windows

Download an FFmpeg static build from the FFmpeg Builds releases page.

Add its bin directory to your Windows PATH.

Then open Command Prompt and run:

ffmpeg -version

Part 10: Download the Model Weights

MuseTalk requires its own model plus several supporting models. The project documents the following model components and folder structure.

Important recommendation:

Do not blindly use download_weights.sh when using the pinned Hugging Face environment.

The automatic download script can cause Hugging Face package version conflicts.

For the most predictable setup, download the files manually.

Step 1: Create the directories

Linux/Vast.ai:

cd /workspace/MuseTalk

mkdir -p models/musetalkV15
mkdir -p models/sd-vae
mkdir -p models/whisper
mkdir -p models/dwpose
mkdir -p models/syncnet
mkdir -p models/face-parse-bisent

Step 2: Download MuseTalk 1.5

Install the Hugging Face CLI if necessary:

python -m pip install "huggingface_hub[cli]==0.30.2"

With Hugging Face Hub 0.30.x, use:

huggingface-cli

rather than the newer:

hf

Download MuseTalk 1.5:

huggingface-cli download TMElyralab/MuseTalk \
  musetalkV15/musetalk.json \
  musetalkV15/unet.pth \
  --local-dir /workspace/MuseTalk/models

Verify:

ls -lh /workspace/MuseTalk/models/musetalkV15/

You should have:

musetalk.json
unet.pth

The unet.pth file is several GB in size.

Step 3: Download the SD VAE

Use an absolute destination path:

huggingface-cli download stabilityai/sd-vae-ft-mse \
  config.json \
  diffusion_pytorch_model.bin \
  diffusion_pytorch_model.safetensors \
  --local-dir /workspace/MuseTalk/models/sd-vae

This avoids the warning about a missing safetensors file when the current code looks for that format.

Your directory should contain:

models/sd-vae/
├── config.json
├── diffusion_pytorch_model.bin
└── diffusion_pytorch_model.safetensors

Step 4: Download Whisper Tiny

huggingface-cli download openai/whisper-tiny \
  config.json \
  pytorch_model.bin \
  preprocessor_config.json \
  --local-dir /workspace/MuseTalk/models/whisper

Expected:

models/whisper/
├── config.json
├── preprocessor_config.json
└── pytorch_model.bin

Step 5: Download DWPose

huggingface-cli download yzd-v/DWPose \
  dw-ll_ucoco_384.pth \
  --local-dir /workspace/MuseTalk/models/dwpose

Expected:

models/dwpose/dw-ll_ucoco_384.pth

Step 6: Download SyncNet

huggingface-cli download ByteDance/LatentSync \
  latentsync_syncnet.pt \
  --local-dir /workspace/MuseTalk/models/syncnet

Expected:

models/syncnet/latentsync_syncnet.pt

Step 7: Download Face Parse

Install gdown:

python -m pip install gdown

Download the Face Parse model:

gdown "https://drive.google.com/uc?id=154JgKpzCPW82qINcVieuPH3fZ2e0P812" \
  -O /workspace/MuseTalk/models/face-parse-bisent/79999_iter.pth

Download ResNet18:

wget -O /workspace/MuseTalk/models/face-parse-bisent/resnet18-5c106cde.pth \
  https://download.pytorch.org/models/resnet18-5c106cde.pth

Part 11: Verify the Model Structure

This step is extremely important.

Run:

find /workspace/MuseTalk/models -type f | sort

The important files should be:

models/dwpose/dw-ll_ucoco_384.pth

models/face-parse-bisent/79999_iter.pth
models/face-parse-bisent/resnet18-5c106cde.pth

models/musetalkV15/musetalk.json
models/musetalkV15/unet.pth

models/sd-vae/config.json
models/sd-vae/diffusion_pytorch_model.bin
models/sd-vae/diffusion_pytorch_model.safetensors

models/syncnet/latentsync_syncnet.pt

models/whisper/config.json
models/whisper/preprocessor_config.json
models/whisper/pytorch_model.bin

Avoid this mistake

Do not accidentally create:

models/models/dwpose/...
models/models/sd-vae/...
models/models/whisper/...

Use absolute paths with --local-dir as shown above.

MuseTalk expects the components directly under models.

Part 12: Prepare Your Input

MuseTalk can use a normal video as the source, and the inference configuration accepts video_path and audio_path. The project recommends 25 FPS video for optimal results.

Create:

mkdir -p /workspace/MuseTalk/input

For a video:

input/video.mp4
input/audio.wav

For image input:

input/person.jpg
input/audio.wav

Part 13: Upload Files to Vast.ai

From your Mac, a simple option is SCP.

Upload video

scp -P 57488 /path/to/video.mp4 \
root@47.186.21.5:/workspace/MuseTalk/input/video.mp4

Upload audio

scp -P 57488 /path/to/audio.wav \
root@47.186.21.5:/workspace/MuseTalk/input/audio.wav

On the Vast server:

ls -lh /workspace/MuseTalk/input/

Part 14: Check the Video

Run:

ffprobe -v error \
-select_streams v:0 \
-show_entries stream=r_frame_rate,width,height \
-of default=noprint_wrappers=1 \
input/video.mp4

For example:

width=1920
height=1080
r_frame_rate=25/1

MuseTalk recommends 25 FPS because that matches its training setup.

If your video is 30 FPS

Convert it:

ffmpeg -i input/video.mp4 \
-r 25 \
-c:v libx264 \
-c:a aac \
input/video_25fps.mp4

Then use:

input/video_25fps.mp4

Part 15: Check the Audio

Run:

ffprobe -v error \
-show_entries format=duration \
-of default=noprint_wrappers=1 \
input/audio.wav

Then:

ffprobe -v error \
-show_entries stream=sample_rate,channels \
-of default=noprint_wrappers=1 \
input/audio.wav

A normal speech file such as:

24000 Hz
1 channel

can work. MuseTalk will process the audio through its speech feature extraction pipeline.

Part 16: Configure Video Inference

Open:

nano configs/inference/test.yaml

Use:

task_0:
  video_path: "input/video.mp4"
  audio_path: "input/audio.wav"

Save with:

Ctrl+O
Enter
Ctrl+X

Part 17: Run MuseTalk 1.5

For Linux/Vast.ai, this explicit command is recommended because it gives control over FP16 and batch size:

python -m scripts.inference \
  --inference_config configs/inference/test.yaml \
  --result_dir results/test \
  --unet_model_path models/musetalkV15/unet.pth \
  --unet_config models/musetalkV15/musetalk.json \
  --version v15 \
  --use_float16 \
  --batch_size 8

The official Linux shortcut is also:

sh inference.sh v1.5 normal

MuseTalk documents the Linux v1.5 normal and v1.5 realtime workflows.

Recommended RTX 4090 settings

Start with:

FP16:       enabled
Batch size: 8
GPU:        RTX 4090 24GB

If you ever encounter CUDA out-of-memory:

--batch_size 4

can be used instead.

Part 18: What a Successful Run Looks Like

A successful run will eventually show:

Starting inference
...
Padding generated images
...
Video generation command: ffmpeg ...
Audio combination command: ffmpeg ...
Results saved to ...

For example:

Results saved to results/test/v15/video_audio.mp4

MuseTalk’s inference workflow performs face preparation, generation, padding, video encoding, and audio combination.

Part 19: Download the Generated Video

From your Mac:

scp -P 57488 \
root@47.186.21.5:/workspace/MuseTalk/results/test/v15/video_audio.mp4 \
~/Downloads/

The video will appear in your Mac’s Downloads folder.

Part 20: Image-to-Video Input

MuseTalk can also accept an image as video_path.

For example:

input/person.jpg
input/audio.wav

then:

task_0:
  video_path: "input/person.jpg"
  audio_path: "input/audio.wav"

The current inference code recognizes common image formats such as:

.jpg
.jpeg
.png
.bmp
.tif
.tiff

The image should ideally contain:

  • One person
  • A clearly visible face
  • Face mostly facing the camera
  • Good lighting
  • Minimal blur
  • Minimal obstruction over the mouth
  • A reasonably large face region

A head-and-shoulders portrait is a good starting point.

Part 21: Image Input Cleanup Fix

Depending on the exact MuseTalk checkout, image input can expose a cleanup bug at the end of inference.

If an image run generates the frames successfully but then fails while deleting a temporary directory, open:

nano scripts/inference.py

Near the beginning of main(args), initialize:

save_dir_full = None

Then find the unconditional cleanup:

shutil.rmtree(save_dir_full)

Replace it with:

if save_dir_full is not None and os.path.isdir(save_dir_full):
    shutil.rmtree(save_dir_full)

Save the file.

Then image inference can be run with:

python -m scripts.inference \
  --inference_config configs/inference/test_image.yaml \
  --result_dir results/image_test \
  --unet_model_path models/musetalkV15/unet.pth \
  --unet_config models/musetalkV15/musetalk.json \
  --version v15 \
  --use_float16 \
  --batch_size 8 \
  --fps 25

Create the image configuration:

nano configs/inference/test_image.yaml

Use:

task_0:
  video_path: "input/person.jpg"
  audio_path: "input/audio.wav"

Part 22: Real-Time Inference

Once normal inference works, you can experiment with the real-time workflow.

Linux:

sh inference.sh v1.5 realtime

MuseTalk’s documentation says to set preparation to True when preparing a new avatar and then use preparation=False when repeatedly generating videos with the same avatar.

This is particularly useful when generating many videos from the same person.

Part 23: Gradio Web Interface

MuseTalk also includes a Gradio interface.

On Vast.ai, launch it on port 7860:

python app.py --use_float16 --port 7860

From your Mac, create the SSH tunnel:

ssh -p 57488 root@47.186.21.5 -L 7860:localhost:7860

Then open in your Mac browser:

http://localhost:7860

Using --use_float16 reduces memory use and inference time. The MuseTalk documentation notes that removing FP16 can improve quality but increases VRAM usage and inference time.

Part 24: Common Errors and Fixes

Error: python: command not found

On a fresh Linux container, check:

python3 --version

Then install/create the Conda Python 3.10 environment and activate:

conda activate MuseTalk

Error: Python 3.14

If:

python --version

shows:

Python 3.14.x

you are still in the base environment.

Run:

conda activate MuseTalk

Then verify:

python --version

You want Python 3.10.

Error: Could not find a version that satisfies torch==2.0.1

Check:

python --version

If you are using Python 3.14, activate the Python 3.10 environment first.

Then run:

python -m pip install \
  torch==2.0.1 \
  torchvision==0.15.2 \
  torchaudio==2.0.2 \
  --index-url https://download.pytorch.org/whl/cu118

Error: ModuleNotFoundError: No module named 'pip' while installing chumpy

Use:

python -m pip install "setuptools<82" wheel

Then:

python -m pip install --no-build-isolation "chumpy==0.70"

Error: Transformers requires huggingface-hub <1.0

Restore:

python -m pip install --force-reinstall "huggingface_hub==0.30.2"

Verify:

python -c "import transformers,huggingface_hub; print(transformers.__version__); print(huggingface_hub.__version__)"

Expected:

4.39.2
0.30.2

Error: hf: command not found

With Hugging Face Hub 0.30.2, use:

huggingface-cli

For example:

huggingface-cli download TMElyralab/MuseTalk ...

Do not upgrade Hugging Face Hub merely to obtain the newer hf command.

Error: opendatalab requires openxlab

After MMLab installation, remove the unnecessary tooling:

python -m pip uninstall -y openmim opendatalab

Then:

python -m pip check

Error: No file named diffusion_pytorch_model.safetensors

Download both VAE formats:

huggingface-cli download stabilityai/sd-vae-ft-mse \
  config.json \
  diffusion_pytorch_model.bin \
  diffusion_pytorch_model.safetensors \
  --local-dir /workspace/MuseTalk/models/sd-vae

Error: Model path is models/models/...

This usually happens when a relative --local-dir is used incorrectly.

Use absolute paths:

--local-dir /workspace/MuseTalk/models/dwpose

rather than:

--local-dir models/dwpose

when working from an unexpected directory.

The correct result is:

models/dwpose/...
models/sd-vae/...
models/whisper/...
models/syncnet/...

not:

models/models/...

Error: CUDA not available

Run:

nvidia-smi

Then:

python -c "import torch; print(torch.cuda.is_available()); print(torch.version.cuda)"

Expected:

True
11.8

Also check:

python -c "import torch; print(torch.cuda.get_device_name(0))"

You should see your NVIDIA GPU.

Error: FFmpeg not found

Linux:

apt-get update
apt-get install -y ffmpeg

Then:

which ffmpeg

Windows:

Make sure FFmpeg’s bin directory is in PATH.

Error: CUDA out of memory

Use FP16:

--use_float16

Lower the batch size:

--batch_size 4

or:

--batch_size 2

A 24GB RTX 4090 is generally a comfortable GPU for MuseTalk inference.

Error: No face detected

Use a source image/video with:

  • One visible person
  • Face relatively large
  • Face facing the camera
  • Good lighting
  • Minimal motion blur
  • No hand covering the face

Part 25: Recommended Vast.ai Workflow

For a clean Vast.ai deployment, the complete process is:

Vast.ai RTX 4090
        ↓
CUDA11 template
        ↓
Miniconda
        ↓
Python 3.10 environment
        ↓
PyTorch 2.0.1 + cu118
        ↓
MuseTalk requirements
        ↓
MMCV 2.0.1
MMDet 3.1.0
MMPose 1.1.0
        ↓
chumpy 0.70 workaround
        ↓
Transformers 4.39.2
HuggingFace Hub 0.30.2
        ↓
FFmpeg
        ↓
MuseTalk 1.5 model weights
        ↓
Video/Image + Audio
        ↓
MuseTalk 1.5 FP16 inference
        ↓
MP4 output

Part 26: The Known-Good Vast AI Environment

For the Vast.ai setup tested with this guide:

GPU:
RTX 4090 24GB

Python:
3.10.x

PyTorch:
2.0.1+cu118

Torchvision:
0.15.2

Torchaudio:
2.0.2

CUDA:
11.8

MMCV:
2.0.1

MMDet:
3.1.0

MMPose:
1.1.0

Chumpy:
0.70

Diffusers:
0.30.2

Transformers:
4.39.2

HuggingFace Hub:
0.30.2

Librosa:
0.11.0

OpenCV:
4.9.0

MuseTalk:
1.5

This combination was successfully used to generate a MuseTalk 1.5 video on an RTX 4090.

Part 27: Quick Start After Everything Is Installed

Once the installation is complete, future sessions only require:

conda activate MuseTalk
cd /workspace/MuseTalk

Then place your:

input/video.mp4
input/audio.wav

Configure:

configs/inference/test.yaml

and run:

python -m scripts.inference \
  --inference_config configs/inference/test.yaml \
  --result_dir results/test \
  --unet_model_path models/musetalkV15/unet.pth \
  --unet_config models/musetalkV15/musetalk.json \
  --version v15 \
  --use_float16 \
  --batch_size 8

Your generated file will normally appear under:

results/test/v15/

Final Recommendation

For the easiest and most stable MuseTalk 1.5 setup on Vast.ai:

GPU: RTX 4090 24GB
Template: CUDA11
Python: 3.10
PyTorch: 2.0.1+cu118
MuseTalk: 1.5
FP16: Enabled
Batch size: 8
Video: 25 FPS preferred

The most important things to avoid are:

Python 3.14
Torch 2.9/2.10+
new xformers
HuggingFace Hub 1.x
isolated chumpy build
models/models/... paths
blindly running the automatic weight script after pinning dependencies

Following the version pins and folder structure above avoids the main dependency and path problems that commonly occur during MuseTalk installation.