Improve SD XL

[patrickvonplaten]

What does this PR do?

This PR makes sure diffusers Stable Diffusion XL can generate images of any size. We also make sure that both single file format and diffusers format can be loaded.

Diffusers format:

from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline
import torch

use_refiner = True

pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, variant="fp16", use_safetensors=True)
pipe.to("cuda")

if use_refiner:
    refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
    refiner.to("cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt=prompt, output_type="latent" if use_refiner else "pil").images[0]

if use_refiner:
    image = refiner(prompt=prompt, image=image[None, :]).images[0]

Single File Format:

from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline
import torch

use_refiner = True

pipe = StableDiffusionXLPipeline.from_single_file("https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/blob/main/sd_xl_base_0.9.safetensors", torch_dtype=torch.float16, use_safetensors=True)
pipe.to("cuda")

if use_refiner:
    refiner = StableDiffusionXLImg2ImgPipeline.from_single_file("https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-0.9/blob/main/sd_xl_refiner_0.9.safetensors", torch_dtype=torch.float16, use_safetensors=True)
    refiner.to("cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt=prompt, output_type="latent" if use_refiner else "pil").images[0]

if use_refiner:
    image = refiner(prompt=prompt, image=image[None, :]).images[0]

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[bghira]

can we pass a scheduler in? i would like to be able to convert it and attach DDIM, as that's a better sampler for SDXL even though it's divergent from their upstream impl.

[JemiloII]

I like the idea of being able to customize the scheduler. I like Euler, but I usually customize the settings of the scheduler.

[bghira]

why is this now defaulting to None? can you explain a bit?

[patrickvonplaten]

It will then later be set to the passed height and width this makes sure we generate correct outputs for different input sizes than 1024

[bghira]

thank you for fixing the CKPT converter.

[Mark-divinci]

i want to increase the batch_size to get larger throughput,but i got linear growth of latency.if i run some incorrect settings?