StableDiffusionControlNetImg2ImgPipeline cannot load CKPT or Safetensor file directly

[thenol]

How to load a ckpt or safetensor file directly using StableDiffusionControlNetImg2ImgPipeline ?

[patrickvonplaten]

Think for now we have to manually convert and load with from_pretrained . More than happy to accept a pull request working on this though!

[0sparsh2]

How to load a ckpt or safetensor file directly using StableDiffusionControlNetImg2ImgPipeline ?

I was trying out a similar pipeline, ControlNetInpaint
Over there the following steps worked for me, you can try the same ->

1. Locally store a safetensors model
2. Link the folder in which it's located
3. Use from_pretrained and use_safetensors = True
   It worked for me

StableDiffusionControlNetInpaintPipeline.from_pretrained(
                "PATH_TO_THE_FOLDER_WITH_SAFETENSORS", controlnet=controlnet_model,
                torch_dtype=torch.float16, use_safetensors=True)

[WhiteWolf47]

Think for now we have to manually convert and load with from_pretrained . More than happy to accept a pull request working on this though!

@patrickvonplaten wdym by "manually convert and load with from_pretrained"? Can you please give an example if i have .ckpt file of the model!

[patrickvonplaten]

@yiyixuxu maybe you can check this as well (should be the same PR / mechanism): #2584

[patrickvonplaten]

And another one: #3722

Seems like we can close three issues with one pull request:

• StableDiffusionControlNetImg2ImgPipeline cannot load CKPT or Safetensor file directly #3907
• StableDiffusionControlNetInpaintPipeline does not seem to support .safetensors? #3974
• StableDiffusionControlNetPipeline.from_ckpt() does not exist #3722

[patrickvonplaten]

And another one: #4052 - wow this is a wanted feature!

[patrickvonplaten]

@yiyixuxu let me take this as you're already on a bunch of other issues :-)

[patrickvonplaten]

Hey @thenol,

Can you check if this works for you: #4084

[EmilioNicolas]

Think for now we have to manually convert and load with from_pretrained . More than happy to accept a pull request working on this though!

Thanks @patrickvonplaten for the PR that's looking great, but meanwhile as I am working with REPLICATE-COG and sticking to new releases of the library, how could I load a CKPT file for a stable_diffusion_controlnet_img2img? Should I follow this? https://huggingface.co/docs/diffusers/v0.18.2/en/using-diffusers/other-formats#pytorch-ckpt

[patrickvonplaten]

Hey @EmilioNicolas,

You can load .ckpt files also with the from_single_file(...) function