[LoRA refactor 2] feat: add support for load_lora().

[sayakpaul]

What does this PR do?

Adds a load_lora() method to UNet2DConditionLoadersMixin so that users can directly load PEFT LoRAs without having to rely on a pipeline class.

TODO

• Documentation
• Tests

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

Suggested change

	PEFT_WEIGHT_NAME = "adapter_model.bin"
	PEFT_WEIGHT_NAME_SAFE = "adapter_model.safetensors"
	PEFT_CONFIG_NAME = "adapter_config.json"
	PEFT_WEIGHT_NAME = "adapter_model.bin"
	PEFT_WEIGHT_NAME_SAFE = "adapter_model.safetensors"
	PEFT_CONFIG_NAME = "adapter_config.json"

@BenjaminBossan @pacman100 @younesbelkada @apolinario - We need to discuss this a bit. I'm all in favor of supporting adapter_config.json mid-term, but we have some peculiarities here:

• Currently LoRAs serialization format is a single file that contains weights for both the Unet and the text_encoder. However, it's also not uncommon that one only trains the unet in which case the single file only contains the unet
• I'd say pretty much always text encoder and unet LoRAs are trained together, but I do see use cases where one might want to load two different LoRA files (one for it's text encoder, one for it's unet)

=> As a consequence the main "loading" function of LoRA is part of the pipeline:

pipeline = DiffusionPipeline.from_pretrained("...")
pipeline.load_lora_weights("...")  # this internally then dispatches to loading the text encoder specific loras to the text encoder and the unet specific loras to the unet

However, it's also very important that we allow loading LoRAs just from the unet or the text encoder:

pipeline.text_encoder.load_lora   # <- here we call Transformers' lora loading function
pipeline.unet.load_lora  # <- this needs to be implemented well in Diffusers

So the question here is, what format and serialization should we use for PEFT that is both:

• Easy to use for training (meaning easy to use just on the unet object when no pipeline is loaded)
• Easy to use for inference (when loaded via pipeline.load_lora_weights)
• Easy to use to mix different LoRAs

Can we allow adapter.json to have a config for either just a text encoder, just a unet, both text encoder and unet
Can adapter.safetensors have both formats?

[BenjaminBossan]

Not sure if I grasped the problem completely, but the 2nd part (separate LoRAs for text encoder and unet) would be the easier one, right? We would just point to different folders, each with their own adapter_config.json etc.

For the combined adapter, we would need to find a new solution, it could be some kind of convention (sub-folder names) or we would need to have a single adapter that takes care of both, but that seems unwieldy and more error prone.

Another use case I'd like to bring up, not sure if it's relevant at this moment: Having a pipeline load multiple different LoRA adapters (or even other type of adapters).

[patrickvonplaten]

but the 2nd part (separate LoRAs for text encoder and unet) would be the easier one, right?

Yes, it'd def be cleaner as a serialization format, but it's very much not the convention right now. At the moment, all LoRAs always only have a single safetensors file. So if we introduce a new 4 file convention (text_encoder's adapter_config.json, text_encoder's adapter_model.safetensors unet's adapter_config.json and unet's adapter_model.safetensors) it might be quite confusing for the community. I'm not too happy about creating a new folder structure as this wasn't super well perceived in the first place.

[pacman100]

I prefer the file format with subfolders for unet/text-encoder each having a adapter_model.safetensors and adapter_config.json. This is exactly mimicing the folder structure of full finetuning/pretrained SD/SDXL in Diffusers just that the config/weights are only for adapters. So, that would be clearly following the standard persistence format of Diffusers

[pacman100]

For portability, I had written a conversion script which would put both the unet and text-encoder weights and their config's in a single safetensors file. The file is this: https://github.com/pacman100/peft-dreambooth/blob/master/peft_utils/merge_peft_sd_ckpt_to_single_safetensors.py

[pacman100]

This way, current paradigm of single safetensors for LoRA would be achieved for PEFT and users are happy. But, this single safetensors file won't work in Auto1111 as it requires changing the state dict keys which is also implemented here https://github.com/pacman100/peft-sd-webui-additional-networks/blob/main/scripts/peft_lora.py

[sayakpaul]

Thanks for your thoughts. But the thing is the diffusion community very much prefers the single file-format at least for LoRAs.

[pacman100]

then these comments https://github.com/huggingface/diffusers/pull/5958/files#r1410149384 and https://github.com/huggingface/diffusers/pull/5958/files#r1410163690 would be the recommended way to have Diffusers specific utils.

[pacman100]

Another thing to keep in mind is that the utils need to generic. For example, A PR to add OFT in PEFT is close to being merged. It follows the standard format of subfolders with adapter+config files. The utils need to be generic to convert both LoRA or OFT into single safetensors format.

[sayakpaul]

Having a utility to output a single file seems to be the best choice here without compromising too much.

[sayakpaul]

@patrickvonplaten I think we can pick this up again now that we're doing #6135. WDYT?

Cc: @pacman100 @younesbelkada @BenjaminBossan

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Not stale.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[kodinkod]

is this planned to be added?
@sayakpaul