Question about the diffuser_DiffSplat module

[MobiusLqm]

Hi, Chenguo, I noticed that your code includes a custom module named diffuser_DiffSplat. Could you let me know which version of the Diffusers library this module is based on or modified from?

Thanks in advance for your clarification!
Qingming

[chenguolin]

Hi @MobiusLqm, I manually modified the latest diffusers library (diffusers==0.32) and tried to comment in detail on codes to clarify modifications. The folder structure is the same as the original repo.

Generally:

• Modifications in diffusers_diffsplat/models are most for (1) "multi-view attention" that gets inputs in (B*V, N, D) then operates the attention operation in (B, V*N, D), (2) a new function from_pretrained_new() for UNet and Transformer that initializes models with different input channels, e.g., 4 (SD latent) in the original Stable Diffusion, while 10 or 11 (RGB + plucker + (optional) binary mask) for our DiffSplat.
• diffusers_diffsplat/pipelines are implemented for each base model in diffusers_diffsplat/models accordingly with some fancy functions (such as Instant3D-style noise initialization), which are however not really used.
• diffusers_diffsplat/schedulers are only for DPM-Solver++ flow matching scheduler, which is copied from the diffusers PR and not really used.
• diffusers_diffsplat/training_utils.py is only for EMAModel that can really set self.use_ema_warmup as described in the diffusers PR.