kolorstext-to-imagediffusion-modelschatglm3sdxlcomputer-visiongenerative-aitutorial

Kolors Guide: Bilingual Photorealistic Diffusion AI

Kwai Kolors Architecture: Photorealistic Text-to-Image Diffusion with ChatGLM3 Text Conditioning

The Problem: The Semantic Bottleneck and Typography Failure of Diffusion Models

In modern enterprise creative suites, automated visual marketing, e-commerce catalog generation, and intelligent multimodal agents powered by AI Fetchly, text-to-image generative models serve as critical visual synthesis engines. Whether generating marketing banners from product descriptions gathered via automated visual content harvesting or synthesizing complex storyboards with real-time metadata extraction, foundation models must faithfully render intricate spatial compositions and legible text directly inside pixels.

However, standard open-source and proprietary text-to-image diffusion backbones (such as Stable Diffusion 1.5, SDXL, and early Midjourney iterations) suffer from systemic architectural bottlenecks that degrade visual utility in production:

1. The 77-Token Context Ceiling of CLIP Encoders

Most modern latent diffusion models rely on pre-trained Contrastive Language-Image Pretraining (CLIP) text encoders (such as OpenAI CLIP ViT-L/14 or OpenCLIP ViT-bigG):

  • Hardcoded Length Constraint: CLIP enforces a strict sequence length ceiling of 77 tokens.
  • Semantic Truncation: If a user provides a detailed prompt describing multiple characters, specific architectural textures, atmospheric lighting, and brand copy (easily exceeding 100 to 200 words), standard diffusion pipelines silently discard all tokens beyond the 77th token.
  • Loss of Fine-Grained Attributes: Detailed background specifications, secondary object placements, and subtle emotional directions are completely dropped before reaching the diffusion denoiser.
The CLIP Semantic Truncation Barrier: User Prompt: "A futuristic cyberpunk street market at night in Tokyo with neon signs displaying 'AI Fetchly' and 'Open Source', crowds of people holding transparent umbrellas under gentle neon rain, reflective wet asphalt, glowing holographic vending machines on the left, a ramen food stall on the right with steam rising into cold ambient blue air..." ┌───────────────────────────────────────────────┬────────────────────────────────┐ │ First 77 Tokens: Processed by CLIP │ Tokens 78+: SILENTLY TRUNCATED!│ │ [Tokyo, street, neon, crowds, umbrellas...] │ [Vending machines, ramen stall,│ │ ✅ Preserved in Text Embedding │ steam rising, ambient blue...]│ │ │ ❌ Completely Ignored by Model! │ └───────────────────────────────────────────────┴────────────────────────────────┘

2. The In-Image Typography and Text Rendering Breakdown

Rendering legible text inside generated images—such as store signs, book covers, street posters, and packaging labels—has historically been the Achilles’ heel of diffusion models.

When prompted to render phrases like “AI Fetchly” or Chinese characters like “快手科技”, standard models produce distorted, illegible glyphs:

  1. Contrastive Training Bias: CLIP was trained with a global contrastive objective between whole image embeddings and whole sentence embeddings. It never learned token-level character-to-pixel spatial alignments.
  2. Subword Tokenizer Blindness: Standard BPE tokenizers collapse common words into single opaque token IDs (e.g., token 14205), depriving the visual denoiser of individual letter or character stroke information.
  3. Monolingual English Dominance: CLIP was trained almost exclusively on English web alt-text. Prompts containing Chinese, Japanese, or multilingual cultural terms trigger severe semantic collapse, forcing international users into awkward translation pre-processing.

3. Weak Compositional Binding and Attribute Confusion

In complex multi-object prompts (e.g., “A red leather jacket on a wooden chair next to a blue ceramic vase with yellow sunflowers”), standard diffusion models frequently suffer from attribute bleeding:

  • The color red bleeds onto the vase, or the sunflowers turn blue.
  • Cross-attention layers in early diffusion backbones lack deep syntactic parse trees, failing to resolve which adjectives modify which nouns when multiple entities share the scene.

Creative professionals were forced to adopt complex multi-stage ControlNet workflows, regional prompt extensions, or Photoshop inpainting to repair mangled text and displaced objects.

The Solution: Kwai Kolors Dual-Stream Diffusion Architecture

To eliminate the semantic bottleneck and achieve photorealistic bilingual synthesis, researchers and engineers from Kuaishou Technology introduced Kolors: Effective Training of Diffusion Model for Photorealistic Text-to-Image Synthesis (arXiv:2407.03158).

Rather than relying on frozen, small-scale CLIP encoders, Kolors re-engineers text understanding by integrating ChatGLM3-6B—a massive, bilingual autoregressive language model—directly into the diffusion conditioning pipeline:

The Kwai Kolors Architectural Paradigm: ┌────────────────────────────────────────────────────────────────────────┐ │ Dual-Stream Photorealistic Diffusion Architecture │ ├───────────────────────────────────┬────────────────────────────────────┤ │ Text Conditioning Stream │ Visual Latent Generation Stream │ ├───────────────────────────────────┼────────────────────────────────────┤ │ • ChatGLM3-6B Autoregressive LLM │ • Latent Diffusion SDXL UNet │ │ • Up to 256 text tokens (3.3x) │ • 1024 × 1024 native resolution │ │ • Deep compositional semantics │ • Pre-trained on billions of pairs │ │ • Bilingual Chinese & English text│ • Flawless in-image typography │ └───────────────────────────────────┴────────────────────────────────────┘

1. Core Principles of Kolors

Kolors achieves state-of-the-art photorealism and compositional alignment through four engineering breakthroughs:

  1. LLM as Text Encoder: By replacing CLIP with ChatGLM3-6B, Kolors expands text context from 77 tokens to 256 tokens, providing rich multi-sentence contextual comprehension, accurate attribute binding, and nuanced stylistic steering.
  2. Bilingual Typography & Character Rendering: ChatGLM3’s tokenizer natively processes character-level tokens and subwords in both Chinese and English. Kolors leverages this fine-grained representation to accurately render complex text, calligraphy, store banners, and logos directly in the image pixels.
  3. Multi-Stage Progressive Visual Training: Trained on billions of image-text pairs across progressive resolutions ($256 \times 256 \rightarrow 512 \times 512 \rightarrow 1024 \times 1024$), utilizing sophisticated aesthetic scoring filters, watermarking removal filters, and human anatomy quality ranking.
  4. Open-Weights Diffusers Integration: Fully compatible with HuggingFace Diffusers (KolorsPipeline), supporting ControlNet, IP-Adapter, and LoRA fine-tuning out of the box.

2. High-Aesthetic Dataset Filtering and Curation Pipeline

A critical differentiator of Kolors is its multi-stage data curation pipeline. While conventional web-scraped image-text datasets (such as LAION-5B) suffer from noisy alt-text, watermarks, compression artifacts, and low-resolution crops, Kuaishou deployed an industrial-scale filtering flywheel:

  • Text Recaptioning with Large Vision-Language Models: Raw web captions are frequently uninformative (e.g., “IMG_4920.JPG” or generic SEO tags). Kolors processed raw images through high-capacity vision-language models to generate rich, descriptive bilingual captions detailing primary subjects, backgrounds, lighting conditions, camera perspectives, and rendered text.
  • Automated Aesthetic Scoring Discriminators: A suite of visual quality classifiers evaluated each candidate image for composition, color harmony, sharpness, and absence of visual noise. Images scoring below strict aesthetic thresholds were pruned, retaining only the top 15% of high-fidelity visual assets.
  • Human Anatomy and Watermarking Scrubbing: Specialized object detectors identified and penalized warped human fingers, distorted facial features, and overlaid corporate watermarks or timestamps, ensuring that the model learned pristine photorealistic priors.

3. SOTA Human Evaluation Benchmarks

In extensive human evaluation studies conducted across thousands of complex prompts:

  • Overall Visual Quality: Evaluators preferred Kolors over Stable Diffusion XL (SDXL) by $64.8%$ vs. $35.2%$, achieving visual fidelity on par with proprietary commercial engines like Midjourney v6.
  • Text Rendering Accuracy: Outperforms SDXL by over $4.2\times$ on in-image text legibility benchmarks, accurately rendering both English words and complex Chinese hanzi characters.
  • Complex Compositional Binding: Achieves over $81.2%$ attribute binding accuracy on multi-subject prompts, minimizing attribute bleeding.

Kolors Architecture: Deep Dive into LLM Conditioning and SDXL UNet Scaling

To understand how Kolors translates long, nuanced natural language prompts into high-resolution visual pixels, we examine its dual-stream conditioning mechanics, cross-attention projection layers, and latent diffusion schedule.

Kolors Detailed Text-to-Image Processing Pipeline: Input Text Prompt: "A vintage cafe storefront with neon sign 'AI Fetchly'..." │ ▼ ChatGLM3-6B Tokenizer (Up to 256 tokens) ┌────────────────────────────────────────────────────────────────────────┐ │ ChatGLM3-6B Autoregressive Transformer Backbone │ │ • Ingests token sequence: X_text in R^(B x L) where L <= 256 │ │ • Extracts penultimate layer hidden states: H_text in R^(B x L x 4096) │ └──────────────────────────────────┬─────────────────────────────────────┘ │ ▼ Linear Feature Projection Module ┌────────────────────────────────────────────────────────────────────────┐ │ Text Projection Layer: Linear(4096 -> 2048) │ │ • Projects LLM representations into SDXL cross-attention dimension │ │ • Context Embedding: C_text in R^(B x L x 2048) │ └──────────────────────────────────┬─────────────────────────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ ▼ Cross-Attention Keys & Values ▼ Timestep t ┌────────────────────────────────────────────────────────────────────────┐ │ Latent Diffusion SDXL UNet Backbone (2.6B Parameters) │ │ │ │ Latent Input: z_t in R^(B x 4 x 128 x 128) (1024x1024 Image Latent) │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Down-Blocks: ResNet + Spatial Transformer (Self & Cross-Attn) │ │ │ │ Mid-Block: Deep Spatial Cross-Attention over C_text │ │ │ │ Up-Blocks: Progressive Upsampling + Skip Connections │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ Predicted Noise: eps_theta(z_t, t, C_text) in R^(B x 4 x 128 x 128) │ └──────────────────────────────────┬─────────────────────────────────────┘ │ Iterative Denoising Loop (30-50 steps) ▼ Denoised Latent: z_0 │ ▼ Autoencoder (SDXL VAE Decoder) Output RGB Image (1024 × 1024)

1. ChatGLM3-6B Text Conditioning Mechanics

In standard SDXL, text conditioning relies on a dual-encoder setup: a CLIP ViT-L encoder ($768$-dim output) concatenated with OpenCLIP ViT-bigG ($1280$-dim output), constrained to $77$ tokens.

Kolors discards the dual CLIP setup entirely in favor of a single, highly expressive language model: ChatGLM3-6B:

  • The input prompt $\mathbf{T}$ is tokenized using ChatGLM3’s sentencepiece vocabulary, supporting sequence lengths up to $L = 256$ tokens.
  • The tokens are processed through ChatGLM3’s 28 transformer layers with rotary position embeddings (RoPE).
  • Kolors extracts the hidden states from the penultimate layer (layer 27), capturing rich semantic representations that combine high-level conceptual themes with fine-grained lexical syntax: $$\mathbf{H}_{\text{text}} \in \mathbb{R}^{B \times L \times 4096}$$
  • A linear projection layer projects $\mathbf{H}{\text{text}}$ from $4,096$ dimensions to $2,048$ dimensions, matching the cross-attention channel width of the SDXL UNet: $$\mathbf{C}{\text{text}} = \mathbf{H}{\text{text}} \mathbf{W}{\text{proj}} \in \mathbb{R}^{B \times L \times 2048}$$

2. Cross-Attention in the SDXL UNet Backbone

The visual latent tensor $\mathbf{z}_t \in \mathbb{R}^{B \times 4 \times 128 \times 128}$ represents the noisy image at diffusion timestep $t$ in the VAE latent space ($8\times$ spatial compression for a $1024 \times 1024$ image).

Inside each spatial transformer block of the UNet, visual features $\mathbf{F}{\text{vis}}$ attend to the projected text embeddings $\mathbf{C}{\text{text}}$ via multi-head cross-attention:

$$\mathbf{Q} = \mathbf{F}_{\text{vis}} \mathbf{W}Q, \quad \mathbf{K} = \mathbf{C}{\text{text}} \mathbf{W}K, \quad \mathbf{V} = \mathbf{C}{\text{text}} \mathbf{W}_V$$

$$\text{CrossAttention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{Softmax}\left(\frac{\mathbf{Q} \mathbf{K}^T}{\sqrt{d_k}}\right) \mathbf{V}$$

Because $\mathbf{C}_{\text{text}}$ spans up to $256$ tokens and carries character-level stroke representations from ChatGLM3, the spatial cross-attention maps can precisely guide the UNet where to form localized high-contrast letter strokes (e.g., rendering the horizontal and vertical lines of letters like “E” and “T” or Chinese characters like “快”).

3. Mathematical Formulation of Classifier-Free Guidance and Latent Denoising

During diffusion training, the model is trained to reverse a forward Gaussian diffusion process that progressively corrupts clean image latents $z_0$ into Gaussian noise over discrete timesteps $t \in [1, T]$:

$$ q(z_t \mid z_0) = \mathcal{N}\left(z_t; \sqrt{\bar{\alpha}_t} z_0, (1 - \bar{\alpha}_t) \mathbf{I} ight) $$

Where $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}t = \prod{s=1}^t \alpha_s$ represents the cumulative noise schedule.

The SDXL UNet $\epsilon_\theta$ is trained to predict the added noise vector by minimizing the mean squared error objective conditioned on the ChatGLM3 text embedding $C_{\text{text}}$:

$$ \mathcal{L}{\text{simple}} = \mathbb{E}{z_0, \epsilon \sim \mathcal{N}(0, \mathbf{I}), t}\left[ \left| \epsilon - \epsilon_\theta(z_t, t, C_{\text{text}}) ight|^2 ight] $$

During inference, to steer visual generation toward strict prompt fidelity while maintaining creative coherence, Kolors applies Classifier-Free Guidance (CFG):

$$ \tilde{\epsilon}\theta(z_t, t, C{\text{text}}) = \epsilon_\theta(z_t, t, \emptyset) + s \cdot \left( \epsilon_\theta(z_t, t, C_{\text{text}}) - \epsilon_\theta(z_t, t, \emptyset) ight) $$

Where $\emptyset$ represents the unconditional null-text embedding and $s$ denotes the guidance scale (typically $s \in [4.5, 6.0]$). Because ChatGLM3-6B produces rich 2048-dimensional context vectors with deep syntactic grounding, the guidance vector $\left( \epsilon_\theta(z_t, t, C_{\text{text}}) - \epsilon_\theta(z_t, t, \emptyset) ight)$ provides exceptionally sharp spatial trajectories toward legible typography and precise object placements.

4. Multi-Stage Progressive Pre-Training Curriculum

Training a text-to-image foundation model with a 6B parameter LLM text encoder requires addressing training stability:

  1. Stage 1 (Low-Resolution Pre-training): Pre-trained on 1 billion image-text pairs at $256 \times 256$ resolution to establish coarse visual layout and semantic correspondence.
  2. Stage 2 (Mid-Resolution Alignment): Trained on 500 million curated pairs at $512 \times 512$ resolution, introducing high-aesthetic filtering and text-dense poster datasets.
  3. Stage 3 (High-Resolution Aesthetic Fine-Tuning): Trained at $1024 \times 1024$ resolution with multi-aspect-ratio bucketing. Images with watermarks, low resolution, or distorted human anatomy were filtered out using automated quality discriminators.
  4. Stage 4 (Direct Preference Optimization - DPO): Fine-tuned with visual DPO on paired human preference ratings, sharpening realistic facial features, hands, skin textures, and typography crispness.

Reference Implementation: Building and Running Kolors with PyTorch and Diffusers

Below is a complete, production-grade implementation demonstrating how to load and execute the Kolors pipeline using PyTorch and HuggingFace Diffusers, alongside a custom module illustrating the ChatGLM3 text conditioning projection.

import torch import torch.nn as nn from typing import Optional, List, Dict from diffusers import KolorsPipeline, EulerDiscreteScheduler class KolorsTextConditioner(nn.Module): """ Simulated Kolors ChatGLM3-6B Text Conditioning Module: Demonstrates extracting hidden representations from a large language model and projecting them into SDXL UNet cross-attention space. """ def __init__(self, llm_dim: int = 4096, unet_dim: int = 2048, max_seq_len: int = 256): super().__init__() self.llm_dim = llm_dim self.unet_dim = unet_dim self.max_seq_len = max_seq_len # Projection layer mapping LLM hidden state to UNet cross-attention dimension self.text_projection = nn.Sequential( nn.Linear(llm_dim, unet_dim), nn.LayerNorm(unet_dim), nn.GELU(), nn.Linear(unet_dim, unet_dim) ) def forward(self, llm_hidden_states: torch.Tensor) -> torch.Tensor: """ Args: llm_hidden_states: Penultimate hidden states from ChatGLM3 (B, L, 4096) Returns: Projected text embeddings ready for UNet cross-attention (B, L, 2048) """ B, L, D = llm_hidden_states.shape assert L <= self.max_seq_len, f"Sequence length {L} exceeds maximum {self.max_seq_len}" projected = self.text_projection(llm_hidden_states) return projected def run_kolors_inference( prompt: str, negative_prompt: str = "", num_inference_steps: int = 35, guidance_scale: float = 5.0, height: int = 1024, width: int = 1024, device: str = "cuda" if torch.cuda.is_available() else "cpu" ): """ Executes native Kwai Kolors text-to-image synthesis using Diffusers KolorsPipeline. """ print(f"[Kolors] Loading Kwai-Kolors/Kolors pipeline on {device}...") # Load pipeline with float16 precision for GPU inference dtype = torch.float16 if device == "cuda" else torch.float32 pipe = KolorsPipeline.from_pretrained( "Kwai-Kolors/Kolors", torch_dtype=dtype, variant="fp16" if dtype == torch.float16 else None ) # Use Euler Discrete Scheduler for fast, sharp generation pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config) pipe = pipe.to(device) # Enable memory-efficient attention (xFormers or FlashAttention) if device == "cuda": pipe.enable_model_cpu_offload() # Reduces peak VRAM to ~12GB! print(f"[Kolors] Generating 1024x1024 image for prompt: '{prompt}'") image = pipe( prompt=prompt, negative_prompt=negative_prompt, height=height, width=width, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale ).images[0] return image

Verification and Sanity Testing of Conditioning Projection

Let us verify our text conditioning module with a synthetic batch to ensure shape compatibility with standard SDXL UNet cross-attention:

# Instantiate Kolors text conditioner conditioner = KolorsTextConditioner(llm_dim=4096, unet_dim=2048, max_seq_len=256) # Simulate ChatGLM3 penultimate hidden states for a 120-token prompt batch_size = 2 prompt_tokens = 120 mock_llm_states = torch.randn(batch_size, prompt_tokens, 4096) # Forward pass through projection layer projected_context = conditioner(mock_llm_states) print(f"[Verification] Input LLM Hidden States: {mock_llm_states.shape}") print(f"[Verification] Projected UNet Context: {projected_context.shape}") assert projected_context.shape == (batch_size, prompt_tokens, 2048), "Dimension mismatch!" print("[Verification] Kolors conditioning dimensions verified successfully!")

Architectural Comparison: Kolors vs. SDXL, FLUX, and Midjourney

To evaluate where Kolors stands in the contemporary generative landscape, we contrast its architecture and empirical benchmarks against three leading models: Stable Diffusion XL (SDXL), FLUX.1 [dev], and Midjourney v6.

1. Qualitative Architecture Matrix

Architectural FeatureStable Diffusion XL (SDXL)FLUX.1 [dev]Midjourney v6Kwai Kolors (2026 SOTA)
Text Encoder BackboneDual CLIP (ViT-L + bigG)T5-XXL + CLIP-LProprietary MultimodalChatGLM3-6B (Autoregressive LLM)
Max Text Context Length77 Tokens256–512 Tokens~100 Words256 Tokens (Over 3x CLIP)
Denoiser ArchitectureLatent UNet (2.6B params)DiT Rectified Flow (12B)Proprietary DiffusionEnhanced SDXL UNet (2.7B)
In-Image Text RenderingPoor (Mangled letters)Very Good (English)Good (English)State-of-the-Art (Bilingual)
Chinese Prompt NuanceNear-zero (English only)LimitedModerateNative State-of-the-Art
Hardware Serving VRAM~8 GB (FP16)~24 GB (FP8/BF16)Cloud API Only~12 GB (FP16 with CPU offload)
Ecosystem CompatibilityNative Diffusers / LoRASpecialized DiTClosed APINative Diffusers / ControlNet

2. Empirical Benchmark Evaluation: Text Rendering, Human Preference, and Composition

On standardized evaluation benchmarks spanning typography accuracy, multi-entity compositional binding, and photorealism:

Bilingual In-Image Text Rendering Accuracy (OCR Recognition Rate %):

  • English Text Rendering:
    • SDXL: $31.4%$ accuracy (frequently drops vowels or duplicates consonants).
    • Midjourney v6: $74.2%$ accuracy.
    • FLUX.1 [dev]: $82.5%$ accuracy.
    • Kwai Kolors: $84.1%$ accuracy (crisp fonts, signage, packaging labels).
  • Chinese Character (Hanzi) Rendering:
    • SDXL: $4.2%$ accuracy (renders random illegible strokes).
    • FLUX.1 [dev]: $28.6%$ accuracy.
    • Kwai Kolors: $87.9%$ accuracy (clean calligraphy, neon signs, banner slogans).

Human Photorealism and Aesthetic Preference (Win-Rate vs. SDXL):

  • On human portraits and natural textures (skin pores, hair strands, fabrics, eyes), Kolors achieves a $68.4%$ win-rate over SDXL.
  • On lighting coherence and reflection realism, evaluators preferred Kolors by $62.1%$.

Production Deployment: Optimization, Multi-Control, and Serving

Deploying Kolors in enterprise production—such as automated e-commerce catalog generation, visual banner design, or digital advertising pipelines—requires balancing the compute footprint of ChatGLM3-6B with high-speed latent denoising.

Production Kolors Serving Pipeline: ┌────────────────────────────────────────────────────────────────────────┐ │ Enterprise Ingestion Gateway (FastAPI / gRPC) │ │ • Validates user prompt length (Max 256 tokens) │ │ • Optional negative prompt & aesthetic keyword injection │ └──────────────────────────────────┬─────────────────────────────────────┘ │ ▼ Asynchronous Dispatch ┌────────────────────────────────────────────────────────────────────────┐ │ High-Throughput Worker (NVIDIA L40S 48GB / RTX 4090 24GB / A100) │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Text Conditioning Engine: ChatGLM3-6B (FP8 / INT8 Quantized) │ │ │ │ • Pre-computes prompt embedding in sub-40ms │ │ │ │ • Output cache for repeated negative prompts & system prefixes │ │ │ └──────────────────────────────────┬─────────────────────────────────┘ │ │ │ Context Embeddings (B x 256 x 2048) │ ┌──────────────────────────────────▼─────────────────────────────────┐ │ │ │ TensorRT-Optimized SDXL UNet Denoiser Engine │ │ │ │ • Fused Multi-Head Attention (FlashAttention-2 / cuDNN) │ │ │ │ • 25-step DPM++ / Euler Discrete Scheduler │ │ │ └──────────────────────────────────┬─────────────────────────────────┘ │ │ │ Denoised Latent (128 x 128 x 4) │ │ ┌──────────────────────────────────▼─────────────────────────────────┐ │ │ │ Tiny VAE / SDXL VAE FP16 Decoder (Sub-15ms) │ │ │ │ • Reconstructs final 1024x1024 RGB Bitmap │ │ │ └────────────────────────────────────────────────────────────────────┘ │ └────────────────────────────────────────────────────────────────────────┘

1. VRAM Optimization and Quantized Serving

Because Kolors couples an SDXL UNet (2.6B) with ChatGLM3-6B (6B), running the pipeline in full float32 would require over 35GB of VRAM.

In production:

  1. ChatGLM3 FP8 / INT8 Quantization: Quantizing ChatGLM3-6B to 8-bit precision (via bitsandbytes or AWQ) reduces text encoder memory footprint from 12GB to under 6.5GB with zero measurable degradation in prompt understanding.
  2. Model CPU Offloading: Using pipe.enable_model_cpu_offload() in Diffusers automatically streams the text encoder into host RAM once text embeddings are computed, freeing GPU VRAM for the UNet denoiser. Total peak VRAM drops to under 12GB, allowing Kolors to run comfortably on consumer RTX 4080 and RTX 4090 GPUs.
  3. TensorRT UNet Compilation: Compiling the SDXL UNet via NVIDIA TensorRT reduces 30-step generation latency from 4.8 seconds down to 1.6 seconds per 1024x1024 image.

2. Multi-Control: ControlNet and IP-Adapter Synergy

Kolors retains the modular architectural benefits of the SDXL family:

  • Kolors-ControlNet: Pre-trained ControlNet weights (Canny, Depth, OpenPose) allow designers to enforce exact human poses and structural wireframes while letting Kolors render photorealistic textures and legible text.
  • Kolors-IP-Adapter: Enables subject-driven generation. By feeding a reference product photo or character face alongside the text prompt, the IP-Adapter preserves character identity across diverse background scenes.

3. Production Best Practices for Text Generation

To maximize text rendering success in production:

  • Enclose Target Text in Quotes: Format target text clearly inside the prompt using quotation marks (e.g., ‘A sign that reads “AI Fetchly”’ or ‘A neon sign written “快手科技”’).
  • Provide Spatial Anchors: State explicitly where the text should appear (e.g., “on a wooden billboard in the top-center” or “printed across the front of the cotton t-shirt”).
  • Calibrate Negative Prompts: Use targeted negative prompts (e.g., “blurry letters, misspelled text, extra characters, distorted typography”) to steer the denoiser toward high-contrast character edges.

By replacing restrictive CLIP encoders with a full-scale bilingual LLM, Kwai Kolors establishes a new state-of-the-art benchmark for open-weights photorealistic image synthesis and in-image typography in 2026.