Generative AI & Image Synthesis

GenAI Image Generation

Sketch-to-Image with Inpainting using Stable Diffusion XL + ControlNet. Turn sketches into photorealistic images and refine them with inpainting — all in a local, interactive Tkinter app.

View Code

Project Overview

The Sketch-to-Image & Inpainting Tool is a local desktop application built with Tkinter that integrates Stable Diffusion XL and ControlNet-Scribble. It converts hand-drawn sketches into high-quality photorealistic images and enables precise inpainting using custom brush masks. The entire pipeline runs on your GPU via Hugging Face Diffusers—ensuring fast, offline generation with complete data privacy.

Key Capabilities

Sketch-to-image generation using SDXL + ControlNet (scribble-based)
Brush-based inpainting on generated images
Real-time inference with multi-threaded preview updates
Prompt customization for subject, design features, and environment
Adjustable brush size, eraser toggle, and undo functionality
One-click image export with JPEG/PNG support

Generation Workflow

1. Sketch Drawing & Pre-processing

You sketch directly on the Tkinter canvas using adjustable brush tools. The sketch is saved as a PIL image, resized to 512×512, and converted into an inverted grayscale control map for ControlNet input—no extra edge-detection required.

2. ControlNet-Scribble Conditioning

The ControlNet model interprets the control map to guide structure-aware generation. The conditioning strength is preconfigured, and generation is tuned for sketch-based inputs using `guess_mode` and dynamic thresholding.

3. SDXL Image Generation

User-defined prompts (subject, style, environment) are combined with tuned negative prompts and passed to `StableDiffusionXLControlNetPipeline`. The model then renders a photorealistic image based on sketch and guidance inputs.

4. Inpainting & Export

Activate Inpainting Mode to paint red mask areas over a generated image. `AutoPipelineForInpainting` selectively regenerates those regions. Results are composited with the base image, previewed in real time, and saved in JPEG/PNG format.

Advanced Features

Sketch-to-Image

Convert rough lines on the Tkinter canvas into photorealistic renders using SDXL + ControlNet-Scribble, faithfully preserving your original strokes.

ControlNet-Scribble Guidance

Uses the xinsir/controlnet-scribble-sdxl model to interpret sketch structure. Conditioning strength is preset for natural control with `guess_mode` and dynamic thresholding.

Interactive Inpainting

Draw red masks over areas to regenerate using SDXL’s inpainting model. Masked regions are selectively updated and seamlessly blended with the original image.

Prompt-Driven Styling

Input fields for subject, design elements, environment, and inpainting prompts enable flexible image generation without relying on separate style models.

Brush & Canvas Control

Adjust brush size, toggle eraser, and undo strokes instantly. Canvas supports real-time sketching, masking, and automatic preview after each stroke.

Offline Export

Save generated or inpainted outputs locally as JPEG or PNG files. Fully offline—no cloud upload or third-party dependencies required.

Generation Examples

Explore various examples of sketch-to-image transformations pre-tuned on a architecture-focused generation.

Note: These examples are pre-tuned on a architecture-focused generation, The model can be pre-tuned on any domain.

Tools & Technologies

Category	Technology
Model	Stable Diffusion XL — RealVisXL V4.0 Lightning (via Hugging Face)
Control Networks	xinsir/controlnet-scribble-sdxl-1.0 for sketch-guided generation
Framework	Hugging Face Diffusers with PyTorch (FP16 + CUDA)
Image Processing	Pillow (PIL) for sketch capture, inpainting masks, and output export
Interface	Tkinter GUI with live drawing, masking, sliders, and prompt inputs
Acceleration	Local GPU (torch.float16 inference, DDIM scheduler)
Model Hub	Pretrained models loaded from Hugging Face Hub
Deployment	Fully offline Python desktop app (no web server required)