MuseTalk Installation: Step by Step for Windows and Linux

Setting up MuseTalk takes a bit of preparation since it depends on Python, PyTorch, CUDA, FFmpeg, and a set of model weights. This guide walks through the full process for both Windows and Linux, in order, so nothing gets missed.

Before You Begin: Requirements

Make sure you have the following ready before starting:

  • Python 3.10 (recommended version)
  • CUDA 11.7 or 11.8 compatible NVIDIA GPU (an NVIDIA Tesla V100 was used for the official real time benchmark of 30fps+, but a laptop RTX 3050 Ti with 4GB VRAM has also been tested successfully in fp16 mode)
  • Conda installed on your system
  • Git installed on your system
  • A stable internet connection (model weights add up to several gigabytes)

Part 1: Build the Python Environment

This part is identical on both Windows and Linux.

Step 1: Create a new conda environment

conda create -n MuseTalk python==3.10

Step 2: Activate the environment

conda activate MuseTalk

Step 3: Install PyTorch 2.0.1

Pick one of these two options.

Using pip:

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

Using conda:

conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia

Step 4: Clone the MuseTalk repository

git clone https://github.com/TMElyralab/MuseTalk.git
cd MuseTalk

Step 5: Install the required Python packages

pip install -r requirements.txt

Step 6: Install the MMLab packages

MuseTalk relies on the MMLab ecosystem for pose and detection tasks:

pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv==2.0.1"
mim install "mmdet==3.1.0"
mim install "mmpose==1.1.0"

Part 2: Set Up FFmpeg

MuseTalk needs FFmpeg for audio and video handling. The setup differs slightly by platform.

Windows

  1. Download the FFmpeg static build from the FFmpeg Builds releases page.
  2. Extract the archive somewhere permanent on your drive.
  3. Add the bin folder inside the extracted ffmpeg directory to your system PATH environment variable.
  4. Confirm it worked by opening Command Prompt and running:
ffmpeg -version

You should see version details printed if it installed correctly.

Linux

  1. Install FFmpeg directly through your package manager:
sudo apt-get install ffmpeg

Or, if you prefer the static build used in official testing, download it from the same FFmpeg Builds releases page and set the path manually:

export FFMPEG_PATH=/path/to/ffmpeg
  1. Confirm the installation:
ffmpeg -version

Part 3: Download the Model Weights

MuseTalk needs several sets of weights before it can run: its own trained weights, plus supporting weights from other open models it depends on (VAE, Whisper, DWPose, and a couple of others).

Option 1: Use the Provided Scripts (Easiest)

Windows:

download_weights.bat

Linux:

sh ./download_weights.sh

This handles all the downloads and folder placement automatically.

Option 2: Download Manually

If you would rather download everything yourself, get these files:

  1. MuseTalk’s own trained weights
  2. sd vae ft mse
  3. whisper tiny
  4. dwpose
  5. syncnet
  6. face parse bisent
  7. resnet18

Once downloaded, place everything inside a models folder in this exact structure:

./models/
├── musetalk
│   └── musetalk.json
│   └── pytorch_model.bin
├── musetalkV15
│   └── musetalk.json
│   └── unet.pth
├── syncnet
│   └── latentsync_syncnet.pt
├── dwpose
│   └── dw-ll_ucoco_384.pth
├── face-parse-bisent
│   ├── 79999_iter.pth
│   └── resnet18-5c106cde.pth
├── sd-vae
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── whisper
    ├── config.json
    ├── pytorch_model.bin
    └── preprocessor_config.json

Getting this folder structure exactly right matters. Most first time setup problems trace back to a misplaced file here.

Part 4: Run Your First Inference

With the environment, FFmpeg, and weights all in place, you’re ready to generate your first output.

Confirm FFmpeg is working

ffmpeg -version

Windows

For a standard (non real time) run using MuseTalk 1.5:

python -m scripts.inference --inference_config configs\inference\test.yaml --result_dir results\test --unet_model_path models\musetalkV15\unet.pth --unet_config models\musetalkV15\musetalk.json --version v15 --ffmpeg_path ffmpeg-master-latest-win64-gpl-shared\bin

For real time inference:

python -m scripts.realtime_inference --inference_config configs\inference\realtime.yaml --result_dir results\realtime --unet_model_path models\musetalkV15\unet.pth --unet_config models\musetalkV15\musetalk.json --version v15 --fps 25 --ffmpeg_path ffmpeg-master-latest-win64-gpl-shared\bin

Linux

For a standard run using MuseTalk 1.5:

sh inference.sh v1.5 normal

For real time inference:

sh inference.sh v1.5 realtime

Note: if you’d rather run MuseTalk 1.0 instead of 1.5, swap v1.5 for v1.0 in the commands above (Linux), or point the Windows command at the models\musetalk folder and pytorch_model.bin file with --version v1 instead of v15.

Part 5: Try the Gradio Web Demo (Optional but Recommended for Beginners)

If typing commands isn’t your preference, MuseTalk ships with a simple browser based interface.

Run this command on either platform (make sure the FFmpeg path matches your own setup):

python app.py --use_float16 --ffmpeg_path ffmpeg-master-latest-win64-gpl-shared\bin

This opens a web page where you can upload a face and audio file and generate output without touching the command line again. Removing --use_float16 gives slightly better quality at the cost of more VRAM and a longer render time.

Common Setup Problems and Fixes

“CUDA not available” or GPU not detected Double check that your installed PyTorch build matches your CUDA version (11.7 or 11.8), and confirm your NVIDIA drivers are current.

FFmpeg not found error This almost always means the FFmpeg path wasn’t added correctly. On Windows, recheck your PATH variable; on Linux, recheck the FFMPEG_PATH export or reinstall through apt-get.

Model weight errors during inference Reopen Part 3 and compare your models folder against the structure shown above, file by file. A single misplaced file is the most common cause.

Out of memory errors Try adding --use_float16 to reduce VRAM use, or lower the resolution/length of your input clip. MuseTalk has been run successfully on a 4GB VRAM laptop GPU in fp16 mode, so most modern GPUs should manage it.