AutoPipeline

AutoPipeline is a task-and-model pipeline that automatically selects the correct pipeline subclass based on the task. It handles the complexity of loading different pipeline subclasses without needing to know the specific pipeline subclass name.

This is unlike DiffusionPipeline, a model-only pipeline that automatically selects the pipeline subclass based on the model.

AutoPipelineForImage2Image returns a specific pipeline subclass, (for example, StableDiffusionXLImg2ImgPipeline), which can only be used for image-to-image tasks.

import torch
from diffusers import AutoPipelineForImage2Image

pipeline = AutoPipelineForImage2Image.from_pretrained(
  "RunDiffusion/Juggernaut-XL-v9", torch_dtype=torch.bfloat16, device_map="cuda",
)
print(pipeline)
"StableDiffusionXLImg2ImgPipeline {
  "_class_name": "StableDiffusionXLImg2ImgPipeline",
  ...
"

Loading the same model with DiffusionPipeline returns the StableDiffusionXLPipeline subclass. It can be used for text-to-image, image-to-image, or inpainting tasks depending on the inputs.

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
  "RunDiffusion/Juggernaut-XL-v9", torch_dtype=torch.bfloat16, device_map="cuda",
)
print(pipeline)
"StableDiffusionXLPipeline {
  "_class_name": "StableDiffusionXLPipeline",
  ...
"

Check the mappings to see whether a model is supported or not.

Trying to load an unsupported model returns an error.

import torch
from diffusers import AutoPipelineForImage2Image

pipeline = AutoPipelineForImage2Image.from_pretrained(
    "openai/shap-e-img2img", torch_dtype=torch.float16,
)
"ValueError: AutoPipeline can't find a pipeline linked to ShapEImg2ImgPipeline for None"

There are three types of AutoPipeline classes, AutoPipelineForText2Image, AutoPipelineForImage2Image and AutoPipelineForInpainting. Each of these classes have a predefined mapping, linking a pipeline to their task-specific subclass.

When from_pretrained() is called, it extracts the class name from the model_index.json file and selects the appropriate pipeline subclass for the task based on the mapping.