DPMSolverMultistepScheduler using karras_sigmas doesn't work anymore

[Bigfield77]

Describe the bug

I had modified the comfyui extension from Zuellni https://github.com/Zuellni/ComfyUI-Custom-Nodes

to enable karras_sigmas for the deepfloyd IF model.

This has stopped working a few months ago, i get an error when enabling karras_sigmas now:
ERROR:root:Traceback (most recent call last):
File "D:\stable\comfyUI\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\stable\comfyUI\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\stable\comfyUI\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\stable\comfyUI\ComfyUI\custom_nodes\ComfyUI-Custom-Nodes\Nodes\IF.py", line 275, in process
images = model(
File "D:\stable\comfyUI\comf\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "D:\stable\comfyUI\comf\lib\site-packages\diffusers\pipelines\deepfloyd_if\pipeline_if.py", line 742, in call
intermediate_images = self.scheduler.step(
File "D:\stable\comfyUI\comf\lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py", line 908, in step
prev_sample = self.multistep_dpm_solver_second_order_update(self.model_outputs, sample=sample, noise=noise)
File "D:\stable\comfyUI\comf\lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py", line 649, in multistep_dpm_solver_second_order_update
self.sigmas[self.step_index + 1],
IndexError: index 101 is out of bounds for dimension 0 with size 101

Prompt executed in 19.44 seconds

It used to make the deepfloyd if generation faster by stopping at step 47 out of a 100. now it seems to try to go further than max step.

Reproduction

//load model:
model = DiffusionPipeline.from_pretrained(
f"DeepFloyd/IF-{class._MODELS[model]}-v1.0",
text_encoder=None,
**class._CONFIG,
)

    if scheduler == "sde-dpmsolver++":
        SdeScheduler = DPMSolverMultistepScheduler.from_config(model.scheduler.config)
        SdeScheduler.config.algorithm_type = 'sde-dpmsolver++'
        model.scheduler= SdeScheduler
            
    if (karrasSigmas) :
        model.scheduler.config.use_karras_sigmas=True

    return self.offload(model, device)

//call generation:
images = model(
prompt_embeds=positive,
negative_prompt_embeds=negative,
height=height,
width=width,
generator=torch.manual_seed(seed),
guidance_scale=cfg,
num_images_per_prompt=batch_size,
num_inference_steps=steps,
callback=callback,
output_type="pt",
).images

Logs

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "D:\stable\comfyUI\ComfyUI\execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "D:\stable\comfyUI\ComfyUI\execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "D:\stable\comfyUI\ComfyUI\execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "D:\stable\comfyUI\ComfyUI\custom_nodes\ComfyUI-Custom-Nodes\Nodes\IF.py", line 275, in process
    images = model(
  File "D:\stable\comfyUI\comf\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\stable\comfyUI\comf\lib\site-packages\diffusers\pipelines\deepfloyd_if\pipeline_if.py", line 742, in __call__
    intermediate_images = self.scheduler.step(
  File "D:\stable\comfyUI\comf\lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py", line 908, in step
    prev_sample = self.multistep_dpm_solver_second_order_update(self.model_outputs, sample=sample, noise=noise)
  File "D:\stable\comfyUI\comf\lib\site-packages\diffusers\schedulers\scheduling_dpmsolver_multistep.py", line 649, in multistep_dpm_solver_second_order_update
    self.sigmas[self.step_index + 1],
IndexError: index 101 is out of bounds for dimension 0 with size 101

System Info

windows 11, cuda, diffusers 0.27.0.dev0

Who can help?

No response

[sayakpaul]

You need to provide a fully reproducible code snippet. From your description it's not clear what can be used to reproduce the error.

Cc: @yiyixuxu

[Bigfield77]

Ok, I will do that in the next few days.

Thanks for your help

[Bigfield77]

Hello, here's the code snippet. if line 51 is commented, it will work fine:

import torch
from accelerate import cpu_offload, cpu_offload_with_hook
from diffusers import DiffusionPipeline
from diffusers import DPMSolverMultistepScheduler
from transformers import BitsAndBytesConfig, T5EncoderModel


device_map = "auto" 

text_encoder = T5EncoderModel.from_pretrained(
	"DeepFloyd/IF-I-M-v1.0",
	subfolder="text_encoder",
	variant="fp16",
	device_map=device_map,
	quantization_config=None,
)

device = "cuda"

textEncoder = DiffusionPipeline.from_pretrained(
	"DeepFloyd/IF-I-M-v1.0",
	text_encoder=text_encoder,
	unet=None,
    variant= "fp16",
    torch_dtype= torch.float16,
    requires_safety_checker= False,
    feature_extractor= None,
    safety_checker= None,
    watermarker= None,
).to("cpu")

#textEncoder= self.offload(textEncoder, device)

model = DiffusionPipeline.from_pretrained(
    f"DeepFloyd/IF-I-XL-v1.0",            
    text_encoder=None,
    #variant= "fp16",
    #torch_dtype= torch.float16,
    requires_safety_checker= False,
    feature_extractor= None,
    safety_checker= None,
    watermarker= None,
).to("cuda")
       

SdeScheduler = DPMSolverMultistepScheduler.from_config(model.scheduler.config)
SdeScheduler.config.algorithm_type = 'sde-dpmsolver++'
model.scheduler= SdeScheduler

        
model.scheduler.config.use_karras_sigmas=True

#model= self.offload(model, device)
positivePrompt="an americana painting of a storm over the colorado mountains"
negativePrompt="blurry"

positive, negative = textEncoder.encode_prompt(
            prompt=positivePrompt,
            negative_prompt=negativePrompt,
)

images = model(
    prompt_embeds=positive,
    negative_prompt_embeds=negative,
    height=64,
    width=64,
    generator=torch.manual_seed(42),
    guidance_scale=7,
    num_images_per_prompt=1,
    num_inference_steps=100,
    output_type="pt",
).images

images = (images - images.min()) / (images.max() - images.min())
images = images.permute(0, 2, 3, 1)
images = images.to("cpu", torch.float32)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Bigfield77]

Commenting to prevent autoclose

[yiyixuxu]

thank you for your patience!
confirm that I can reproduce this error and looking into it now

[Bigfield77]

Thanks! :)

[yiyixuxu]

@Bigfield77

I fixed it in #7577 so it would work without an error
however, I'm not sure IF works with DPM schedulers. I was under the impression that it does not. can you share with us if you've tested it out?

[Bigfield77]

@yiyixuxu
thanks for the fix! I will test it later tonight.

It was working before with DPM scheduler on an earlier version of diffusers

[yiyixuxu]

It was working before with DPM scheduler on an earlier version of diffusers

ok good to know! it would work then