docs/VACE.md

VACE ControlNet Guide

VACE is a powerful ControlNet that enables Video-to-Video and Reference-to-Video generation. It allows you to inject your own images into output videos, animate characters, perform inpainting/outpainting, and continue videos.

Overview

VACE is probably one of the most powerful Wan models available. With it, you can:

• Inject people or objects into scenes
• Animate characters
• Perform video inpainting and outpainting
• Continue existing videos
• Transfer motion from one video to another
• Change the style of scenes while preserving depth

Getting Started

Model Selection

1. Select either "Vace 1.3B" or "Vace 13B" from the dropdown menu
2. Note: VACE works best with videos up to 7 seconds with the Riflex option enabled

Input Types

VACE accepts three types of visual hints (which can be combined):

1. Control Video

• Transfer motion or depth to a new video
• Use only the first n frames and extrapolate the rest
• Perform inpainting with grey color (127) as mask areas
• Grey areas will be filled based on text prompt and reference images

2. Reference Images

• Use as background/setting for the video
• Inject people or objects of your choice
• Select multiple reference images
• Tip: Replace complex backgrounds with white for better object integration
• Always describe injected objects/people explicitly in your text prompt

3. Video Mask

• Stronger control over which parts to keep (black) or replace (white)
• Perfect for inpainting/outpainting
• Example: White mask except at beginning/end (black) keeps first/last frames while generating middle content

Common Use Cases

Motion Transfer

Goal: Animate a character of your choice using motion from another video Setup:

• Reference Images: Your character
• Control Video: Person performing desired motion
• Text Prompt: Describe your character and the action

Object/Person Injection

Goal: Insert people or objects into a scene Setup:

• Reference Images: The people/objects to inject
• Text Prompt: Describe the scene and explicitly mention the injected elements

Character Animation

Goal: Animate a character based on text description Setup:

• Control Video: Video of person moving
• Text Prompt: Detailed description of your character

Style Transfer with Depth

Goal: Change scene style while preserving spatial relationships Setup:

• Control Video: Original video (for depth information)
• Text Prompt: New style description

Integrated Matanyone Tool

WanGP includes the Matanyone tool, specifically tuned for VACE workflows. This helps create control videos and masks simultaneously.

Creating Face Replacement Masks

1. Load your video in Matanyone
2. Click on the face in the first frame
3. Create a mask for the face
4. Generate both control video and mask video with "Generate Video Matting"
5. Export to VACE with "Export to current Video Input and Video Mask"
6. Load replacement face image in Reference Images field

Advanced Matanyone Tips

• Negative Point Prompts: Remove parts from current selection
• Sub Masks: Create multiple independent masks, then combine them
• Background Masks: Select everything except the character (useful for background replacement)
• Enable/disable sub masks in Matanyone settings

Recommended Settings

Quality Settings

• Skip Layer Guidance: Turn ON with default configuration for better results
• Long Prompts: Use detailed descriptions, especially for background elements not in reference images
• Steps: Use at least 15 steps for good quality, 30+ for best results

Sliding Window Settings

For very long videos, configure sliding windows properly:

• Window Size: Set appropriate duration for your content
• Overlap Frames: Long enough for motion continuity, short enough to avoid blur propagation
• Discard Last Frames: Remove at least 4 frames from each window (VACE 1.3B tends to blur final frames)

Background Removal

VACE includes automatic background removal options:

• Use for reference images containing people/objects
• Don't use for landscape/setting reference images (first reference image)
• Multiple background removal types available

Window Sliding for Long Videos

Generate videos up to 1 minute by merging multiple windows:

How It Works

• Each window uses corresponding time segment from control video
• Example: 0-4s control video → first window, 4-8s → second window, etc.
• Automatic overlap management ensures smooth transitions

Settings

• Window Size: Duration of each generation window
• Overlap Frames: Frames shared between windows for continuity
• Discard Last Frames: Remove poor-quality ending frames
• Add Overlapped Noise: Reduce quality degradation over time

Formula

Generated Frames = [Windows - 1] × [Window Size - Overlap - Discard] + Window Size

Multi-Line Prompts (Experimental)

• Each line of prompt used for different window
• If more windows than prompt lines, last line repeats
• Separate lines with carriage return

Advanced Features

Extend Video

Click "Extend the Video Sample, Please!" during generation to add more windows dynamically.

Noise Addition

Add noise to overlapped frames to hide accumulated errors and quality degradation.

Frame Truncation

Automatically remove lower-quality final frames from each window (recommended: 4 frames for VACE 1.3B).

External Resources

Official VACE Resources

• GitHub: https://github.com/ali-vilab/VACE/tree/main/vace/gradios
• User Guide: https://github.com/ali-vilab/VACE/blob/main/UserGuide.md
• Preprocessors: Gradio tools for preparing materials

Recommended External Tools

• Annotation Tools: For creating precise masks
• Video Editors: For preparing control videos
• Background Removal: For cleaning reference images

Troubleshooting

Poor Quality Results

1. Use longer, more detailed prompts
2. Enable Skip Layer Guidance
3. Increase number of steps (30+)
4. Check reference image quality
5. Ensure proper mask creation

Inconsistent Windows

1. Increase overlap frames
2. Use consistent prompting across windows
3. Add noise to overlapped frames
4. Reduce discard frames if losing too much content

Memory Issues

1. Use VACE 1.3B instead of 13B
2. Reduce video length or resolution
3. Decrease window size
4. Enable quantization

Blurry Results

1. Reduce overlap frames
2. Increase discard last frames
3. Use higher resolution reference images
4. Check control video quality

Tips for Best Results

1. Detailed Prompts: Describe everything in the scene, especially elements not in reference images
2. Quality Reference Images: Use high-resolution, well-lit reference images
3. Proper Masking: Take time to create precise masks with Matanyone
4. Iterative Approach: Start with short videos, then extend successful results
5. Background Preparation: Remove complex backgrounds from object/person reference images
6. Consistent Lighting: Match lighting between reference images and intended scene