README.md

Paint-With-Words

Paper | Demo

Paint-with-words is a method proposed by researchers from NVIDIA that allows users to control the location of objects by selecting phrases and painting them on the canvas. The user-specified masks increase the value of corresponding entries of the attention matrix in the cross-attention layers.

Inspired by this method, we created a simple a1111-style sketching UI that allows multi-mask input to address same area on different tokens. Also, textual-inversion and LoRA support are fully functional*, you can add them to the generation process and adjust the strength and area they are applied to.

Config and Run

1. Set your model path in

   sd-paint-with-words/app.py

   Lines 28 to 35 in 15e800e

   	models = [
   	# format: name, model_path, clip_skip
   	("AbyssOrangeMix_Base", "/root/workspace/storage/models/orangemix", 2),
   	("Stable Diffuison 1.5", "/root/models/stable-diffusion-v1-5", 1),
   	("AnimeSFW", "/root/workspace/animesfw", 2),
   	]
   	base_name, base_model, clip_skip = models[0]

2. python app.py

Some samples

Sketch	Image