ironjr/semantic-draw

SemanticDraw is a real-time interactive text-to-image generation framework that allows you to draw with meanings 🧠 using semantic brushes 🖌️.

---

🚀 Quick Start

# Install
conda create -n semdraw python=3.12 && conda activate semdraw
git clone https://github.com/ironjr/semantic-draw
cd semantic-draw
pip install -r requirements.txt
Run streaming demo
cd demo/stream
python app.py --model "runwayml/stable-diffusion-v1-5" --port 8000
Open http://localhost:8000 in your browser

For SD3 support, additionally run:

pip install git+https://github.com/initml/diffusers.git@clement/feature/flash_sd3

Note: this is default in requirements.txt

---

📚 Table of Contents

• Features
• Installation
• Demo Applications
• Usage Examples
• Documentation
• FAQ
• Citation

⭐ Features

| Interactive Drawing | Prompt Separation | Real-time Editing | | :---: | :---: | :---: | | | | | | Paint with semantic brushes | No unwanted content mixing | Edit photos in real-time |

---

🔧 Installation

Basic Installation

conda create -n smd python=3.12 && conda activate smd
git clone https://github.com/ironjr/StreamMultiDiffusion
cd StreamMultiDiffusion
pip install -r requirements.txt

Stable Diffusion 3 Support

pip install git+https://github.com/initml/diffusers.git@clement/feature/flash_sd3

---

🎨 Demo Applications

We provide several demo applications with different features and model support:

1. StreamMultiDiffusion (Main Demo)

Real-time streaming interface with semantic drawing capabilities.

cd demo/stream
python app.py --model "your-model" --height 512 --width 512 --port 8000

2. Semantic Palette

Simplified interface for different SD versions:

SD 1.5 Version

cd demo/semantic_palette
python app.py --model "runwayml/stable-diffusion-v1-5" --port 8000

SDXL Version

cd demo/semantic_palette_sdxl
python app.py --model "your-sdxl-model" --port 8000

SD3 Version

cd demo/semantic_palette_sd3
python app.py --port 8000

---

💻 Usage Examples

Python API

import torch
from model import StableMultiDiffusionPipeline
Initialize
device = torch.device('cuda:0')
smd = StableMultiDiffusionPipeline(device, hf_key='runwayml/stable-diffusion-v1-5')
Generate
image = smd.sample('A photo of the dolomites')
image.save('output.png')

import torch
from model import StableMultiDiffusionPipeline
from util import seed_everything
Setup
seed_everything(2024)
device = torch.device('cuda:0')
smd = StableMultiDiffusionPipeline(device)
Define prompts and masks
prompts = ['background: city', 'foreground: a cat', 'foreground: a dog']
masks = load_masks()  # Your mask loading logic
Generate
image = smd(prompts, masks=masks, height=768, width=768)
image.save('output.png')

from model import StreamMultiDiffusion
Initialize streaming pipeline
smd = StreamMultiDiffusion(device, height=512, width=512)
Register layers
smd.update_single_layer(idx=0, prompt='background', mask=bg_mask)
smd.update_single_layer(idx=1, prompt='object', mask=obj_mask)
Stream generation
while True:
    image = smd()
    display(image)

Jupyter Notebooks

Explore our notebooks directory for interactive examples:

• Basic usage tutorial
• Advanced region control
• SD3 examples
• Custom model integration

📖 Documentation

Detailed Guides

• Old README
• Notebooks

Paper

For technical details, see our paper and project page.

---

🙋 FAQ

---

🚩 Recent Updates

• 🔥 June 2025: Presented at CVPR 2025
• ✅ June 2024: SD3 support with Flash Diffusion
• ✅ April 2024: StreamMultiDiffusion v2 with responsive UI
• ✅ March 2024: SDXL support with Lightning LoRA
• ✅ March 2024: First version released

---

🌏 Citation

@inproceedings{lee2025semanticdraw,
    title="{SemanticDraw:} Towards Real-Time Interactive Content Creation from Image Diffusion Models",
    author={Lee, Jaerin and Jung, Daniel Sungho and Lee, Kanggeon and Lee, Kyoung Mu},
    booktitle={CVPR},
    year={2025}
}

---

🤗 Acknowledgements

Built upon StreamDiffusion, MultiDiffusion, and LCM. Special thanks to the Hugging Face team and the model contributors.

---

📧 Contact

Please email jarin.lee@gmail.com or open an issue.