add text-encoder part in train_text_to_image_lora.py

[MikeHanKK]

Is your feature request related to a problem? Please describe.
The train_text_to_image_lora.py example is still using the old lora training format that only includes the UNET part.

Describe the solution you'd like
It would be nice to add the text-encoder part of training and to be inferenced with the load_lora_weights method.

Describe alternatives you've considered
It seems the train_dreambooth_lora.py example has already integrated the text-encoder part and has started the new lora format. But somehow the lora models I trained with the diffusers dreambooth example are not very good. It looks like text-to-images training often yields better results.

[patrickvonplaten]

Is it common to also train the text encoder for text to image fine-tuning cc @sayakpaul ? Not sure how important this use case is at the moment

[sayakpaul]

Yeah, for full fine-tuning, the text encoder is not usually trained. Also, for full fine-tuning, the assumption is that you're using a little more than at least 500 samples; otherwise, it's very easy to overfit.

I guess to mitigate that overfitting (due to the lesser number of samples), DreamBooth introduced prior-reservation, a neat way to induce regularization.

So, if you have lesser number of samples, I'd suggest starting with DreamBooth or even Custom Diffusion (without the text encoder training). You should also maybe try experimenting with different schedulers to see how far you push the performance that way.

Hope this helps.

[abhijitpal1247]

@sayakpaul can we have a parameter like "scale" for text encoder lora weights too? In that way we can control how much the fine-tuned lora layers can affect the generation that can help when the lora layers overfit. If the layers overfit we can reduce their influence using scale.

[sayakpaul]

Sure!

Going forward please either follow conversations in similar issues or open a new issue thread as your question is not really related to this current issue.

Having a separation of concerns helps us better prioritise and better maintain things :)

[sayakpaul]

@MikeHanKK if it's okay with you, could we close the issue?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.