[LoRA] feat: support non-diffusers lumina2 LoRAs.

[sayakpaul]

What does this PR do?

Fixes:
#10866

Cc: @nitinmukesh

Results (under the same seed)

LoRA	No LoRA

Code

import torch
from diffusers import Lumina2Text2ImgPipeline
from peft.tuners.tuners_utils import BaseTunerLayer

pipe = Lumina2Text2ImgPipeline.from_pretrained(
    "Alpha-VLLM/Lumina-Image-2.0", torch_dtype=torch.bfloat16
)

# Art Style of Hitoshi Ashinano https://civitai.com/models/1269546/art-style-of-hitoshi-ashinano-lumina-image-20
pipe.load_lora_weights("newgenai79/lumina2_lora",weight_name="Art_Style_of_Hitoshi_Ashinano.safetensors")
pipe.enable_model_cpu_offload()

prompt = "Hitoshi Ashinano style A young girl with vibrant green hair and large purple eyes peeks out from behind a white wooden door. She is wearing a white shirt and have a curious expression on her face. The background shows a blue sky with a few clouds, and there's a white fence visible. Green leaves hang down from the top left corner, and a small white circle can be seen in the sky. The scene captures a moment of innocent curiosity and wonder."
image = pipe(
    prompt, 
    negative_prompt="blurry, ugly, bad, deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, cropped, out of frame, worst quality, low quality, jpeg artifacts, fused fingers, morbid, mutilated, extra fingers, mutated hands, bad anatomy, bad proportion, extra limbs", 
    guidance_scale=6,
    num_inference_steps=35, 
    generator=torch.manual_seed(0)
).images[0]
image.save("lumina2_lora.png")

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

For https://github.com/huggingface/diffusers/actions/runs/13544909753/job/37854074489?pr=10909, I have opened #10911

For 2nd and third failures, pinged @DN6 internally. I don't think they are triggered by the changes of this PR.

Fourth one seems Hub-related.

Any objections to going ahead with merging?

[nitinmukesh]

Awesome, thank you @sayakpaul .

Looking forward to getting it merged.

[sayakpaul]

@DN6 the failing tests seem to be unrelated to this PR?

[DN6]

Is this prefix specific to Lumina? Should we always just remove it?

[sayakpaul]

It is not specific to Lumina2 but specific to external trainer libraries. In all the past iterations where we have supported non-diffusers LoRA checkpoints, we have removed it because it's not in the diffusers compatible format.

We are not removing the prefix. We are using it to detect if the state dict is non-diffusers. If so, we are converting the state dict.

This is how rest of the non-diffusers checkpoints across different models have been supported in diffusers.