fix bugs for sd3 controlnet training

[xduzhangjiayu]

What does this PR do?

This PR is for fix some bugs for sd3 controlnet training

1. fix issue for multi-gpu training (examples\controlnet\train_controlnet_sd3.py)
2. fix issue for enabling --gradient_checkpointing in training process (src\diffusers\models\controlnet_sd3.py)

Who can review?

@yiyixuxu @sayakpaul @asomoza

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Thanks, looks good to me.

In a future PR, I will add a few more tests related to gradient checkpointing.

[xduzhangjiayu]

Hi, any comment on this PR ? @yiyixuxu

[yiyixuxu]

thanks!

[tobiasfshr]

Hi, I have a question on 1: Is having unused parameters expected behavior? The following parameters seem to be unused:

transformer_blocks.11.ff_context.net.2.bias, transformer_blocks.11.ff_context.net.2.weight, transformer_blocks.11.ff_context.net.0.proj.bias, transformer_blocks.11.ff_context.net.0.proj.weight, transformer_blocks.11.attn.to_add_out.bias, transformer_blocks.11.attn.to_add_out.weight

[Zheng-Fang-CH]

sformer_blo

Hi, I meet the same error, do you solve it now?

[xduzhangjiayu]

Hi, I have never met this error, could you please describe in detail, thanks!

[Zheng-Fang-CH]

Hi, I have never met this error, could you please describe in detail, thanks!

Hi, I didn't change the code in diffusers, but the weight parameters in controlnets seem not be used to compute the final loss, but obviously, these weights have been initialized.

[xduzhangjiayu]

Hi, I have never met this error, could you please describe in detail, thanks!

Hi, I didn't change the code in diffusers, but the weight parameters in controlnets seem not be used to compute the final loss, but obviously, these weights have been initialized.

Seems maybe I have met before, but I'm not sure, could you please contact me with email: xduzhangjiayu@163.com for further discussion ?