[Community] reference only control

[okotaku]

Refer to Mikubill/sd-webui-controlnet#1236

Reference Image

Output Image of reference_attn=True and reference_adain=False

Output Image of reference_attn=False and reference_adain=True

Output Image of reference_attn=True and reference_adain=True

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[jfischoff]

Should we include the group norm modification in this PR as well: https://github.com/Mikubill/sd-webui-controlnet/pull/1278/files#diff-8c8d004eed5a3078434f6fbde15c178e472565ebfcb3119f308f9292c8eb7514R458 ?

Right now using the group norm or reference_adain+attn gives the best results for reference only. I would definitely like to see this get added either in this PR or a subsequent one.

[patrickvonplaten]

Works for me! Thanks!

[patrickvonplaten]

Think there is one merge conflict that we need to resolve & then we can get this one merged :-)

[kadirnar]

Will you add controlnet support? How can we use it with controlnet? @okotaku

[wangdong-ivymobile]

It doesn't work for me for EulerAncestralDiscreteScheduler

[Laidawang]

When i run the example i get the error：
AttributeError: 'StableDiffusionReferencePipeline' object has no attribute 'image_processor'

[wangdong-ivymobile]

When i run the example i get the error： AttributeError: 'StableDiffusionReferencePipeline' object has no attribute 'image_processor'

You can't use the version from PyPi, you need to pull master branch and install from local (that or wait for next release).

[Laidawang]

When i run the example i get the error： AttributeError: 'StableDiffusionReferencePipeline' object has no attribute 'image_processor'

You can't use the version from PyPi, you need to pull master branch and install from local (that or wait for next release).

thanks, i see

[okotaku]

@kadirnar I will add controlnet version after this PR merged.
@patrickvonplaten I added reference adain. Please review it too.

[okotaku]

@wangdong-ivymobile Thank you for your report. I fixed this bug on latest commit.

[jfischoff]

Just fyi, @lllyasviel fixed a bug recently: Mikubill/sd-webui-controlnet#1309

[kadirnar]

@kadirnar I will add controlnet version after this PR merged.
@patrickvonplaten I added reference adain. Please review it too.

This is great 💯 Can we add inpaint feature like in this repo?

https://github.com/Mikubill/sd-webui-controlnet

[okotaku]

@jfischoff Thank you for your suggestion. I fixed style fidelity rule on latest commit, it is based on Mikubill/sd-webui-controlnet#1309 .

[jfischoff]

@okotaku I really appreciate the work you are doing!

[wangdong-ivymobile]

I tried to use this with multiple images (with slight modification), the result is very bad. I wonder if there are some tricks to make it better?
What I did is basically instead of using a latent of a single image, I concat all the ref image latent together.
What I wish to achieve is maybe to have different ref images contributing differently to the result image, e.g. the first setting up the tone and theme, the second add some style and details, etc.

[wangdong-ivymobile]

Also I tried to use multi controlnet with reference on Webui, the result is also not good but different, I wonder maybe the mechanism is different?

[jfischoff]

I'm getting slightly different results with this pipeline vs the automatic1111.
This pipeline

Automatic1111

Looking at the code I notice the implementation is different between the original and this PR. I'm not sure it is responsible for the differences, but I was curious for instance why some of the changes were made.

For instance the original code using the style blend to control adding sequences to the bank

                if outer.attention_auto_machine == AutoMachine.Write:
                    if outer.attention_auto_machine_weight > self.attn_weight:
                        self.bank.append(self_attention_context.detach().clone())
                        self.style_cfgs.append(outer.current_style_fidelity)
                if outer.attention_auto_machine == AutoMachine.Read:
                    if len(self.bank) > 0:
                        style_cfg = sum(self.style_cfgs) / float(len(self.style_cfgs))
                        self_attn1_uc = self.attn1(x_norm1, context=torch.cat([self_attention_context] + self.bank, dim=1))
                        self_attn1_c = self_attn1_uc.clone()
                        if len(outer.current_uc_indices) > 0 and style_cfg > 1e-5:
                            self_attn1_c[outer.current_uc_indices] = self.attn1(
                                x_norm1[outer.current_uc_indices],
                                context=self_attention_context[outer.current_uc_indices])
                        self_attn1 = style_cfg * self_attn1_c + (1.0 - style_cfg) * self_attn1_uc
                    self.bank = []
                    self.style_cfgs = []
                if self_attn1 is None:
                    self_attn1 = self.attn1(x_norm1, **context=self_attention_context)**

Will this PR does the check during the read mode:

                if MODE == "write":
                    self.bank.append(norm_hidden_states.detach().clone())
                    attn_output = self.attn1(
                        norm_hidden_states,
                        encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
                        attention_mask=attention_mask,
                        **cross_attention_kwargs,
                    )
                if MODE == "read":
                    if attention_auto_machine_weight > self.attn_weight:
                        attn_output_uc = self.attn1(
                            norm_hidden_states,
                            encoder_hidden_states=torch.cat([norm_hidden_states] + self.bank, dim=1),
                            # attention_mask=attention_mask,
                            **cross_attention_kwargs,
                        )
                        attn_output_c = attn_output_uc.clone()
                        if do_classifier_free_guidance and style_fidelity > 0:
                            attn_output_c[uc_mask] = self.attn1(
                                norm_hidden_states[uc_mask],
                                encoder_hidden_states=norm_hidden_states[uc_mask],
                                **cross_attention_kwargs,
                            )
                        attn_output = style_fidelity * attn_output_c + (1.0 - style_fidelity) * attn_output_uc
                        self.bank.clear()
                    else:
                        attn_output = self.attn1(
                            norm_hidden_states,
                            encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
                            attention_mask=attention_mask,
                            **cross_attention_kwargs,
                        **)

I haven't looked closely enough to see if there are other differences, nor do I know if this is why my images are different. I'm just curious mostly.

[skirsten]

I believe this should be

noise = randn_tensor(ref_image_latents.shape, generator=generator, device=ref_image_latents.device, dtype=ref_image_latents.dtype)

from the utils import to ensure the generation is deterministic.

[jfischoff]

@okotaku that makes sense. Thanks for answer my question.

[patrickvonplaten]

Cool let's merge this one!

[learningyan]

hi. TypeError: Transformer2DModel.forward() got an unexpected keyword argument 'attention_mask'
also got an unexpected keyword argument 'encoder_attention_mask'
How to solve it ? @okotaku

[okotaku]

@learningyan #3508
I fixed the bug in this PR.
Please pull latest main branch.

[learningyan]

@okotaku
i have pulled latest main branch, still have this question in line 501 and 502 of examples/community/stable_diffusion_controlnet_reference.py

when I comment out these codes, the code is right. but the quality of generated images is poor. So I'm not sure if commenting out these codes is correct.

reference image :

inference code:

pipe = StableDiffusionReferencePipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
safety_checker=None,
torch_dtype=torch.float16
).to('cuda:0')

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

result_img = pipe(ref_image=input_image,
prompt="1girl, masterpiece, best quality",
negative_prompt="lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry",
num_inference_steps=20,
num_images_per_prompt=1,
reference_attn=True,
reference_adain=True).images[0]
result_img.save('reference.png')

result_img:

the results in sd-webui-controlnet (see Mikubill/sd-webui-controlnet#1236)

So How can I get the similar result as sd-webui-controlnet?
thanks a lot.

[okotaku]

@learningyan

i have pulled latest main branch, still have this question in line 501 and 502 of examples/community/stable_diffusion_controlnet_reference.py

You need to update diffusers too.

git pull
pip install .

the quality of generated images is poor.

You can change base model.
For example,

pipe = StableDiffusionReferencePipeline.from_pretrained(
"andite/anything-v4.0",
safety_checker=None,
torch_dtype=torch.float16
).to('cuda:0')

And this result is reference attention only,

result_img = pipe(ref_image=input_image,
prompt="1girl, masterpiece, best quality",
negative_prompt="lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry",
num_inference_steps=20,
num_images_per_prompt=1,
reference_attn=True,
reference_adain=False,
style_fidelity=0.5  # you can set style_fidelity=1.0
).images[0]

Mikubill/sd-webui-controlnet#1236 (comment)

When examining the details more closely, it appears that the actual model being used is anythingv3 and a custom model named animevae.pt, etc.

[vijishmadhavan]

why I am getting kind of white tint on images using controlnet reference only?

`
pipe = StableDiffusionControlNetReferencePipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1",
controlnet=controlnet,
safety_checker=None,
torch_dtype=torch.float16
).to('cuda:0')

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) image1 = Image.fromarray(image)
image = np.array(image)
low_threshold = 100
high_threshold = 200
canny = cv2.Canny(image, low_threshold, high_threshold)
canny = canny[:, :, None]
canny = np.concatenate([canny, canny, canny], axis=2)
canny_image = Image.fromarray(canny)
result_img = pipe(ref_image=image1,
prompt=prompt,
negative_prompt="lowres, two head, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry",
image=canny_image,
num_inference_steps=20,
controlnet_conditioning_scale=0.8,
reference_attn=True,
reference_adain=True,
style_fidelity=0.5).images[0]`

[miiiz]

i just git from the lastest diffusers and install following the important note but got this error. StableDiffusionReferencePipeline works fine.

AttributeError: 'StableDiffusionControlNetReferencePipeline' object has no attribute '_default_height_width'

[blx0102]

@miiiz you can copy the code of that function from reference file to the controlnet reference file

[okotaku]

@miiiz @blx0102 Thank you for your feedback.
I updated the codes.

#3723

[alexisrolland]

Hi there, this thread is a bit long and confusing. Is there any documentation somewhere that describes the classes StableDiffusionReferencePipeline and StableDiffusionCobtrolNetReferencePipeline?

[gasvn]

@okotaku Thanks for your diffusers implementation. Based on your code, I achieve a cross-image region drag based on the reference scheme. 1. Use inpaint controlnet to extract inpainted region feature from another image. 2. Use the segment-anything controlnet to keep reasonable pose.

Code in https://github.com/sail-sg/EditAnything

[okotaku]

@gasvn Really cool projects! Thank you for your contribution.

[ethankongee]

two questions here:

1. I see StableDiffusionControlNetReferencePipeline and StableDiffusionReferencePipeline. What are the differences?
2. I'm unable to import StableDiffusionControlNetReferencePipeline directly from the diffusers module. From the doc here, looks like I should use custom_pipeline kwarg?

[patrickvonplaten]

Yes you need to use the custom_pipeline kwarg :-)

[djj0s3]

two questions here:

1. I see StableDiffusionControlNetReferencePipeline and StableDiffusionReferencePipeline. What are the differences?
2. I'm unable to import StableDiffusionControlNetReferencePipeline directly from the diffusers module. From the doc here, looks like I should use custom_pipeline kwarg?

For anyone coming to this later, you'll definitely need custom_pipeline like so:

controlnet_reference_pipe = StableDiffusionPipeline.from_pretrained(
  MODEL_ID,
  cache_dir=MODEL_CACHE_DIR,
  custom_pipeline="stable_diffusion_controlnet_reference",
  controlnet=CONTROLNET_PIPE,
  safety_checker=None,
  torch_dtype=torch.float16
  )

[amrakm]

@djj0s3 does that mean you can't use reference-only with other ControlNets using StableDiffusionMultiControlNetPipeline

[djj0s3]

@amrakm are you trying to use reference-only as a controlnet to pass into another pipe with other controlnets? Interesting idea! One option you can try is to import this class directly and use it as StableDiffusionControlNetReferencePipeline.from_pretrained(...):...Since it is written as a subclass of StableDiffusionControlNetPipeline, you should be able to accomplish what you're trying to do. I'm coincidentally loading it this way vs using custom_pipeline since I had to tweak the class to get a couple of other things I'm doing working with it. I haven't tried what you're doing, but that approach should work for you.

[Akmpfen]

I see the code has reference_attn and reference_adain, but how to use reference_only?

[dotieuthien]

why I am getting kind of white tint on images using controlnet reference only?

` pipe = StableDiffusionControlNetReferencePipeline.from_pretrained( "stabilityai/stable-diffusion-2-1", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16 ).to('cuda:0')

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) image1 = Image.fromarray(image) image = np.array(image) low_threshold = 100 high_threshold = 200 canny = cv2.Canny(image, low_threshold, high_threshold) canny = canny[:, :, None] canny = np.concatenate([canny, canny, canny], axis=2) canny_image = Image.fromarray(canny) result_img = pipe(ref_image=image1, prompt=prompt, negative_prompt="lowres, two head, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry", image=canny_image, num_inference_steps=20, controlnet_conditioning_scale=0.8, reference_attn=True, reference_adain=True, style_fidelity=0.5).images[0]`

I still face with the same issue. Could you please give me any advices? @okotaku

[chyohoo]

why I am getting kind of white tint on images using controlnet reference only?
pipe = StableDiffusionControlNetReferencePipeline.from_pretrained( "stabilityai/stable-diffusion-2-1", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16 ).to('cuda:0') pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) image1 = Image.fromarray(image) image = np.array(image) low_threshold = 100 high_threshold = 200 canny = cv2.Canny(image, low_threshold, high_threshold) canny = canny[:, :, None] canny = np.concatenate([canny, canny, canny], axis=2) canny_image = Image.fromarray(canny) result_img = pipe(ref_image=image1, prompt=prompt, negative_prompt="lowres, two head, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry", image=canny_image, num_inference_steps=20, controlnet_conditioning_scale=0.8, reference_attn=True, reference_adain=True, style_fidelity=0.5).images[0]

I still face with the same issue. Could you please give me any advices? @okotaku

normalized the reference image into [-1,1] not [0,1]

[patrickvonplaten]

@DN6 there is still lots of interest I think. Wonder if we should circle back to a naive diffusers integration as proposed in #4257

[animebing]

@okotaku @patrickvonplaten the ref_image in stable_diffusion_controlnet_reference.py is not normalized to [-1, 1], because control_image_processor in StableDiffusionControlNetPipeline does not do this, but this normalization is needed in prepare_ref_latents when using VAE