[Enhance] Update reference

[okotaku]

Update reference pipeline based on diffusers update.

12a232e
74fd735
5990014
bf16a97

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[atomassoni]

Not 100% sure I loaded this correctly but I am no longer getting the error that's on diffusers main, this is hopefully from this branch

pipe = DiffusionPipeline.from_pretrained(model_path,
                       custom_pipeline="/content/diffusers/examples/community/stable_diffusion_controlnet_reference.py",
                       controlnet=canny, 
                       safety_checker=None, 
                       torch_dtype=torch.float16
                ).to('cuda:0')

result_img = pipe(prompt=prompt ,
   image=canny_edged_image,
   ref_image=image_local,
   num_inference_steps=20,
   reference_attn=True,
   reference_adain=True).images[0]

pipe loads fine, and I'm getting an error on inference

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 36>:38                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115 in decorate_context       │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /root/.cache/huggingface/modules/diffusers_modules/local/stable_diffusion_controlnet_reference.p │
│ y:237 in __call__                                                                                │
│                                                                                                  │
│   234 │   │   assert reference_attn or reference_adain, "`reference_attn` or `reference_adain`   │
│   235 │   │                                                                                      │
│   236 │   │   # 1. Check inputs. Raise error if not correct                                      │
│ ❱ 237 │   │   self.check_inputs(                                                                 │
│   238 │   │   │   prompt,                                                                        │
│   239 │   │   │   image,                                                                         │
│   240 │   │   │   callback_steps,                                                                │
│                                                                                                  │
│ /content/diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet.py:548 in check_inputs │
│                                                                                                  │
│    545 │   │   │   or is_compiled                                                                │
│    546 │   │   │   and isinstance(self.controlnet._orig_mod, ControlNetModel)                    │
│    547 │   │   ):                                                                                │
│ ❱  548 │   │   │   self.check_image(image, prompt, prompt_embeds)                                │
│    549 │   │   elif (                                                                            │
│    550 │   │   │   isinstance(self.controlnet, MultiControlNetModel)                             │
│    551 │   │   │   or is_compiled                                                                │
│                                                                                                  │
│ /content/diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet.py:630 in check_image  │
│                                                                                                  │
│    627 │   │   │   prompt_batch_size = prompt_embeds.shape[0]                                    │
│    628 │   │                                                                                     │
│    629 │   │   if image_batch_size != 1 and image_batch_size != prompt_batch_size:               │
│ ❱  630 │   │   │   raise ValueError(                                                             │
│    631 │   │   │   │   f"If image batch size is not 1, image batch size must be same as prompt   │
│    632 │   │   │   )                                                                             │
│    633                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: If image batch size is not 1, image batch size must be same as prompt batch size. image batch size: 
600, prompt batch size: 1

[patrickvonplaten]

Cool that looks good to me!

[patrickvonplaten]

Good to merge from my side - @okotaku let me know if you'd like this PR to be merged

[okotaku]

@patrickvonplaten LGTM!
Thank you.