Add model card with metadata, links, and sample usage

README.md

@@ -1,3 +1,62 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+pipeline_tag: image-to-video
+library_name: diffusers
+---
+
+# Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
+
+This repository contains the `CogVideoControlNetModel` for **Sketch3DVE**, a novel sketch-based 3D-aware video editing method. Sketch3DVE enables detailed local manipulation of videos, even with significant viewpoint changes, by handling novel view content consistency, preserving unedited regions, and translating sparse 2D sketch inputs into realistic 3D video outputs.
+
+The model was presented in the paper:
+[Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing](https://huggingface.co/papers/2508.13797)
+
+Project Page and Code: [http://geometrylearning.com/Sketch3DVE/](http://geometrylearning.com/Sketch3DVE/)
+
+## Abstract
+Recent video editing methods achieve attractive results in style transfer or appearance modification. However, editing the structural content of 3D scenes in videos remains challenging, particularly when dealing with significant viewpoint changes, such as large camera rotations or zooms. Key challenges include generating novel view content that remains consistent with the original video, preserving unedited regions, and translating sparse 2D inputs into realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a sketch-based 3D-aware video editing method to enable detailed local manipulation of videos with significant viewpoint changes. To solve the challenge posed by sparse inputs, we employ image editing methods to generate edited results for the first frame, which are then propagated to the remaining frames of the video. We utilize sketching as an interaction tool for precise geometry control, while other mask-based image editing methods are also supported. To handle viewpoint changes, we perform a detailed analysis and manipulation of the 3D information in the video. Specifically, we utilize a dense stereo method to estimate a point cloud and the camera parameters of the input video. We then propose a point cloud editing approach that uses depth maps to represent the 3D geometry of newly edited components, aligning them effectively with the original 3D scene. To seamlessly merge the newly edited content with the original video while preserving the features of unedited regions, we introduce a 3D-aware mask propagation strategy and employ a video diffusion model to produce realistic edited videos. Extensive experiments demonstrate the superiority of Sketch3DVE in video editing.
+
+## Sample Usage
+This model is a ControlNet component designed to be used with a compatible base video generation pipeline within the Hugging Face `diffusers` library. You would typically load this ControlNet model and then integrate it into a `DiffusionPipeline` for video-to-video editing tasks.
+
+Here's a conceptual example of how you might load this ControlNet model:
+
+```python
+from diffusers import ControlNetModel
+import torch
+
+# Load the Sketch3DVE ControlNet model
+# Replace "your-repo-id/Sketch3DVE" with the actual model ID if different
+controlnet = ControlNetModel.from_pretrained("your-repo-id/Sketch3DVE", torch_dtype=torch.float16)
+
+# This ControlNet would then be integrated into a larger video diffusion pipeline.
+# For example, with a hypothetical `ImageToVideoControlNetPipeline`:
+# from diffusers import ImageToVideoControlNetPipeline, UNet3DConditionModel, AutoencoderKL, DDIMScheduler, CLIPTextModel, CLIPTokenizer
+#
+# # Load a compatible base video generation model (e.g., a fine-tuned text-to-video model)
+# # You would need to replace these with actual compatible base model components.
+# unet = UNet3DConditionModel.from_pretrained("path/to/base/video/unet")
+# vae = AutoencoderKL.from_pretrained("path/to/base/video/vae")
+# text_encoder = CLIPTextModel.from_pretrained("path/to/base/video/text_encoder")
+# tokenizer = CLIPTokenizer.from_pretrained("path/to/base/video/tokenizer")
+# scheduler = DDIMScheduler.from_pretrained("path/to/base/video/scheduler")
+#
+# pipeline = ImageToVideoControlNetPipeline(
+#     vae=vae,
+#     text_encoder=text_encoder,
+#     tokenizer=tokenizer,
+#     unet=unet,
+#     controlnet=controlnet, # Integrate the loaded ControlNet
+#     scheduler=scheduler,
+# )
+# pipeline.to("cuda")
+#
+# # You can then use the pipeline to generate video based on a sketch image and text prompt.
+# # For example:
+# # output_video = pipeline(
+# #     prompt="A car driving on a road",
+# #     image=your_sketch_image, # Your input sketch image
+# #     num_inference_steps=50
+# # ).images[0]
+```
+This snippet demonstrates the loading of the ControlNet model. The complete usage within a video generation pipeline will depend on the specific base video model it is intended to be paired with.