Background and Layout Evaluation

[aaghawaheed]

Am I doing something wrong? I am unable to generate a harmonized layout

Background Training

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export INSTANCE_DIR="./../data/background/train"
export OUTPUT_DIR="./../logs/background"

accelerate launch train_background.py
--pretrained_model_name_or_path=$MODEL_NAME
--train_text_encoder
--instance_data_dir=$INSTANCE_DIR
--output_dir=$OUTPUT_DIR
--resolution=512
--train_batch_size=1
--gradient_checkpointing
--learning_rate=1e-5
--lr_scheduler="constant"
--lr_warmup_steps=500
--num_train_epochs=50
--resume_from_checkpoint=latest
--with_spatial_loss
--checkpointing_steps=10000

Background Evaluation Results

• FID: 25.15
• CLIP: 17.36
• Saliency Ratio: 0.1756

Layout Training

python main.py --dataset webui --exp layout
--data_dir ../data
--epoch 100 --lr 1.5e-5 --lr_decay
--encode_backbone swin --encode_embd 1024
--finetune_vb --pretrain_vb --debug

Layout Evaluation Results

• Occlusion: 10.86
• Alignment: 0.28
• Overlap: 10.19

[aaghawaheed]

Also in the following command, I can not see the usage of "generator_path" param

python pipeline.py
--prompt "Rose Valentines' Day"
--mode "background"
--encoder_path /path/to/encoder
--decoder_path /path/to/decoder
--generator_path logs/background-ours

[whaohan]

For the layout generation training, remove the --debug option to use the training set, otherwise only valset is used for training.

[aaghawaheed]

Thanks for the response. Let me try

[aaghawaheed]

Evaluation Script

EXP=swin
DATASET=webui
COMMAND=category_generate
python main.py --encode_backbone swin --encode_embd 1024
--dataset $DATASET --exp $EXP --evaluate
--decoder_path ../logs/$DATASET/$EXP/checkpoints/decoder.pth
--encoder_path ../logs/$DATASET/$EXP/checkpoints/encoder.pth
--eval_command $COMMAND
--calculate_harmony
--save_pkl

python eval.py webui logs/$DATASET/$EXP/generated_layout_$COMMAND.pth

Training Script

python main.py --dataset webui --exp layout
--data_dir ../data
--epoch 100 --lr 1.5e-5 --lr_decay
--encode_backbone swin --encode_embd 1024
--finetune_vb --pretrain_vb

1. Could you please also fix the the --exp folder, as the name is different in both the training and evaluation scripts?
2. What's the difference between harmony and coverage?
3. ["category_generate", "real_image", "reconstruction"]—what's's the difference between them?
4. Where can I find the presentation generation script.

Thank you for your assistance with my questions. I appreciate your support and look forward to your response.

[aaghawaheed]

Training Script

python main.py --dataset webui --exp layout
--data_dir ../data
--epoch 100 --lr 1.5e-5 --lr_decay
--encode_backbone swin --encode_embd 1024
--finetune_vb --pretrain_vb

Evaluation Command

EXP=layout
DATASET=webui
COMMAND=category_generate
python main.py --encode_backbone swin --encode_embd 1024
--dataset $DATASET --exp $EXP --evaluate
--decoder_path /logs/$DATASET/$EXP/checkpoints/decoder_99.pth
--encoder_path /logs/$DATASET/$EXP/checkpoints/encoder_99.pth
--eval_command $COMMAND
--calculate_harmony
--save_pkl
--save_image

python eval.py webui /logs/$DATASET/$EXP/generated_layout_$COMMAND.pth

Evaluation Results

using device: cuda:0
load iou data
train <dataset.WebUI object at 0x7f2806174c40>
load iou data
train dataset vocab_size: 231
train dataset max_length: 42
encoder status: name-swin, grad-True, pretrain-False
/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392036766/work/aten/src/ATen/native/TensorShape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
testing...
dataset length: 393
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 393/393 [01:36<00:00, 4.08it/s]
harmony is: 0.17446137576242893
Dataset: webui
Alignment: 0.22
Overlap: 17.67

Still the results are not satisfactory, am I doing something wrong?