FQiao/GenStereo-sd2.1

[ICCV 2025] Towards Open-World Generation of Stereo Images and Unsupervised Matching

This repository contains the model presented in Towards Open-World Generation of Stereo Images and Unsupervised Matching. The models are finetuned on Stable Diffusion v2.1.

Abstract

Stereo images are fundamental to numerous applications, including extended reality (XR) devices, autonomous driving, and robotics. Unfortunately, acquiring high-quality stereo images remains challenging due to the precise calibration requirements of dual-camera setups and the complexity of obtaining accurate, dense disparity maps. Existing stereo image generation methods typically focus on either visual quality for viewing or geometric accuracy for matching, but not both. We introduce GenStereo, a diffusion-based approach, to bridge this gap. The method includes two primary innovations (1) conditioning the diffusion process on a disparity-aware coordinate embedding and a warped input image, allowing for more precise stereo alignment than previous methods, and (2) an adaptive fusion mechanism that intelligently combines the diffusion-generated image with a warped image, improving both realism and disparity consistency. Through extensive training on 11 diverse stereo datasets, GenStereo demonstrates strong generalization ability. GenStereo achieves state-of-the-art performance in both stereo image generation and unsupervised stereo matching tasks.

How to use

Environment

We tested our codes on Ubuntu with nVidia A100 GPU. If you're using other machines like Windows, consider using Docker. You can either add packages to your python environment or use Docker to build an python environment. Commands below are all expected to run in the root directory of the repository.

We tested the environment with python >=3.10 and CUDA =11.8. To add mandatory dependencies run the command below.

pip install -r requirements.txt

To run developmental codes such as the example provided in jupyter notebook and the live demo implemented by gradio, add extra dependencies via the command below.

pip install -r requirements_dev.txt

Download pretrained models

GenStereo uses pretrained models which consist of both our finetuned models and publicly available third-party ones. Download all the models to checkpoints directory or anywhere of your choice. You can do it manually or by the download_models.sh script.

Download script

bash scripts/download_models.sh 

Manual download

Models and checkpoints provided below may be distributed under different licenses. Users are required to check licenses carefully on their behalf.

1. Our finetuned models, we provide two versions of GenStereo
   • v1.5: 512px, faster, model card.
   • v2.1: 768px, better performance, high resolution, takes more time, model card.
2. Pretrained models:
   • sd-vae-ft-mse
     • download config.json and diffusion_pytorch_model.safetensors to checkpoints/sd-vae-ft-mse
   • sd-image-variations-diffusers
     • download image_encoder/config.json and image_encoder/pytorch_model.bin to checkpoints/image_encoder
3. MDE (Monocular Depth Estimation) models
   • We use Depth Anything V2 as the MDE model and get the disparity maps. The final checkpoints directory must look like this:

.
├── depth_anything_v2_vitl.pth
├── genstereo-v1.5
│   ├── config.json
│   ├── denoising_unet.pth
│   ├── fusion_layer.pth
│   ├── pose_guider.pth
│   └── reference_unet.pth
├── genstereo-v2.1
│   ├── config.json
│   ├── denoising_unet.pth
│   ├── fusion_layer.pth
│   ├── pose_guider.pth
│   └── reference_unet.pth
├── image_encoder
│   ├── config.json
│   └── pytorch_model.bin
└── sd-vae-ft-mse
    ├── config.json
    └── diffusion_pytorch_model.safetensors

Inference

You can easily run the inference code by running the following command, and the results will be save under ./vis folder.

python test.py /path/to/your/image

Gradio live demo

An interactive live demo is also available. Start gradio demo by running the command below, and goto http://127.0.0.1:7860/ If you are running it on the server, be sure to forward the port 7860.

Or you can just visit Spaces hosted by Hugging Face to try it now.

python app.py

Train

Please read Train_Guide.md.

Citation

  @inproceedings{qiao2025genstereo,
    author    = {Qiao, Feng and Xiong, Zhexiao and Xing, Eric and Jacobs, Nathan},
    title     = {Towards Open-World Generation of Stereo Images and Unsupervised Matching},
    booktitle = {Proceedings of the {IEEE/CVF} International Conference on Computer Vision ({ICCV})},
    year      = {2025},
    eprint    = {2503.12720},
    archiveprefix = {arXiv},
    primaryclass = {cs.CV}
  }

Acknowledgements

Our codes are based on GenWarp, Moore-AnimateAnyone and other repositories. We thank the authors of relevant repositories and papers.