internlm-xcomposervision-language-modelsmultimodalvideo-reasoninglong-contextshanghai-ai-labtutorial

InternLM-XComposer2.5 Guide: 96-Image & Video Reasoning

October 6, 2026

InternLM-XComposer2.5 Architecture: Long-Context Multi-Image and Video Understanding

The Multi-Image & Video Barrier in Modern Vision-Language AI

In modern enterprise automation, automated web research, and multimodal intelligence workflows powered by AI Fetchly, visual comprehension must extend far beyond single isolated photographs. Real-world tasks require ingesting dense sequences: multi-page PDF contracts, multi-step website screenshot trajectories, slide presentations, and hour-long continuous video recordings.

However, despite rapid advancements in Vision-Language Models (VLMs) like LLaVA, CogVLM, and early InternVL iterations, deploying open-weights models on multi-image and video workloads reveals three critical architectural roadblocks:

1. The Multi-Image Token Explosion

Standard Vision-Language Models rely on a rigid visual tokenization pipeline:

Each incoming image is passed through a Vision Transformer (ViT) and projected into fixed feature grids (e.g., $576$ or $2,304$ tokens per image).
When a user feeds an 80-page quarterly earnings disclosure or a 50-step browsing trajectory into the model, the visual token count alone explodes past 100,000 to 200,000 tokens.
Because self-attention complexity scales quadratically $\mathcal{O}(N^2)$ and Key-Value (KV) cache memory scales linearly $\mathcal{O}(N)$, feeding more than 5 to 10 images instantly exhausts GPU High-Bandwidth Memory (HBM), triggering catastrophic Out-of-Memory (OOM) crashes.

+-----------------------------------------------------------------------------------+
|               The Conventional VLM Multi-Image Breakdown                          |
|                                                                                   |
|  [Multi-Page Document / 50 Web Screenshots]                                       |
|           |                                                                       |
|           v (Fixed High-Resolution Tokenization: 2,304 tokens per image)          |
|  +-----------------------------------------------------------------------------+  |
|  | Image 1: 2,304 Tokens                                                        |  |
|  | Image 2: 2,304 Tokens                                                        |  |
|  | Image 3: 2,304 Tokens                                                        |  |
|  | ...                                                                         |  |
|  | Image 50: 2,304 Tokens                                                       |  |
|  | Total Visual Tokens: 115,200 Tokens!                                         |  |
|  +-----------------------------------------------------------------------------+  |
|           |                                                                       |
|           v                                                                       |
|  [Result: Massive KV Cache blowout (>80GB VRAM), severe attention dilution,       |
|           and prompt failure on standard 24GB/40GB enterprise GPUs!]               |
+-----------------------------------------------------------------------------------+

2. Temporal Blindness in Long Video Analysis

Video is inherently sequential, capturing dynamic causal relationships across time:

Standard multimodal models ingest video by sampling a tiny set of sparse keyframes (e.g., 8 to 16 uniformly spaced frames across an entire video).
Slicing a 30-minute surveillance clip or educational lecture into only 16 frames drops over 99.9% of the temporal data.
Brief but crucial events—such as a specific product price flashing on screen for 2 seconds, an algorithmic glitch in a web UI recording, or a rapid hand gesture—fall between sampled frames and become completely invisible.

3. The Resolution vs. Global Context Trade-Off

When engineering teams attempt to reduce visual tokens by downsampling images:

Squeezing multi-page document images into coarse $224 \times 224$ patches renders small typography, table footnotes, and financial decimal points illegible.
Conversely, naive high-resolution patch tiling floods the context window with repetitive background tokens.

When building autonomous data harvesting agents or multimodal RAG engines via an integrated API platform, systems require an architecture capable of reading dozens of high-definition images or thousands of video frames without memory saturation.

The Solution: InternLM-XComposer2.5

Developed by researchers at Shanghai AI Lab, InternLM-XComposer2.5 resolves the multi-image and video bottleneck through a purpose-built long-context multimodal architecture.

Built on an extended 96k-token context window, InternLM-XComposer2.5 natively supports:

Up to 96 interleaved high-resolution images in a single conversational prompt.
Hour-long continuous video feeds (up to 4,000 video frames) with dense temporal coherence.
Dynamic Visual Slicing that dynamically adapts token budgets based on aspect ratios.
Hierarchical Spatiotemporal Token Pooling for long video sequences.

With a compact 7-billion parameter footprint, InternLM-XComposer2.5 rivals closed frontier models like GPT-4V and Gemini 1.5 Pro across 28 leading multimodal benchmarks, setting new open-weights state-of-the-art records on Video-MME (75.4%), DocVQA, and MathVista.

+---------------------------------------------------------------------------------------+
|                 InternLM-XComposer2.5 Multimodal Architecture                         |
|                                                                                       |
|   Input: Up to 96 Images OR 4,000 Video Frames + Long Text Prompt                     |
|                               |                                                       |
|                               v                                                       |
|   1. Dynamic Visual Slicing & Adaptive Tiling                                         |
|   - Evaluates native aspect ratios (width/height) without distortion                  |
|   - Dynamically allocates 1 to 5 local patches only where detail is required          |
|   - Produces compressed visual token representations                                  |
|                               |                                                       |
|                               v                                                       |
|   2. Hierarchical Spatiotemporal Token Pooling (For Video Sequences)                  |
|   - Groups adjacent frames into temporal clusters                                     |
|   - Aggregates inter-frame redundancy while preserving motion vectors                 |
|   - Compresses 4,000 frames into manageable 24k-96k sequence tokens                   |
|                               |                                                       |
|                               v                                                       |
|   3. InternLM2-7B Backbone with 96k Context Window                                    |
|   - Extended Rotary Position Embeddings (RoPE)                                        |
|   - FlashAttention-2 & Grouped-Query Attention (GQA)                                  |
|   - Seamless causal cross-modal reasoning across text, images, and video              |
|                               |                                                       |
|                               v                                                       |
|   [ Output: High-Precision Long-Context Multimodal Reasoning ]                        |
+---------------------------------------------------------------------------------------+

The Three Architectural Pillars of InternLM-XComposer2.5

1. Dynamic Visual Slicing (Aspect-Ratio Aware Tiling)

Rather than forcing all images into a uniform square patch grid, InternLM-XComposer2.5 calculates the optimal grid decomposition based on the image’s original aspect ratio:

Standard square photos are processed as a single global overview plus a $2 \times 2$ grid.
Tall vertical smartphone screenshots or wide panoramic web pages are sliced into $1 \times 4$ or $4 \times 1$ linear tiles.
By matching native proportions, the model avoids zero-padding waste and preserves high-frequency text characters without stretching.

2. Hierarchical Spatiotemporal Video Compression

To ingest up to 4,000 frames from a continuous video recording, InternLM-XComposer2.5 applies hierarchical pooling across time:

The video stream is divided into uniform 1-second clips.
A temporal pooling operator averages static background tokens across adjacent frames while preserving dynamic foreground motion embeddings.
This reduces video token counts by over 85%, allowing an entire 45-minute technical lecture to fit within the 96k context window.

3. Unified Long-Context Multimodal Alignment

Trained on extensive multi-turn conversational datasets, book-length document analyses, and hour-long video QA pairs, the model exhibits robust needle-in-a-haystack retrieval across both textual and visual modalities, accurately pinpointing a visual detail embedded in the 78th image of a prompt.

Empirical Benchmark Comparison

The following table summarizes performance across leading video and multi-image benchmarks:

Model Architecture	Total Parameters	Context Window	Video-MME (Overall)	Video-MME (Long Video >30m)	DocVQA	MVBench
LLaVA-NeXT-Video	7.0 B	32k	52.3%	41.2%	78.4%	55.4%
Qwen-VL-Max	Proprietary	Closed	71.2%	65.4%	93.2%	69.8%
GPT-4V (OpenAI)	Closed	128k	73.8%	67.5%	88.4%	68.5%
Gemini 1.5 Pro	Closed	1M+	75.0%	71.2%	93.1%	70.2%
InternLM-XComposer2.5	7.0 B (Open)	96k	75.4% (SOTA)	72.1% (SOTA)	94.8%	73.5%

InternLM-XComposer2.5 outperforms GPT-4V and Gemini 1.5 Pro on long-video understanding (72.1%), establishing a new state-of-the-art among sub-10B models. For organizations deploying intelligent visual pipelines, this enables enterprise-grade accuracy on single commodity GPUs, verifiable against modern cloud GPU pricing models.

Hands-On Implementation: PyTorch Architecture & Inference Pipeline

To understand how dynamic slicing and spatiotemporal pooling operate, let us build the foundational PyTorch modules: the Dynamic Visual Slicer, the Spatiotemporal Video Pooler, and an end-to-end multi-image inference pipeline.

Step 1: Aspect-Ratio Aware Dynamic Visual Slicer

The following PyTorch module determines optimal slice configurations and tiles images into native sub-crops:

import torch
import torch.nn as nn
import torch.nn.functional as F
from PIL import Image
import torchvision.transforms as T
from typing import List, Tuple

class DynamicVisualSlicer(nn.Module):
    """
    Aspect-ratio aware dynamic visual slicer for high-resolution images.
    Decomposes arbitrary images into 1 to 5 optimal sub-crops plus 1 global overview.
    """
    def __init__(self, tile_size: int = 336, max_slices: int = 5):
        super().__init__()
        self.tile_size = tile_size
        self.max_slices = max_slices
        self.transform = T.Compose([
            T.ToTensor(),
            T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        ])

    def find_best_grid(self, original_width: int, original_height: int) -> Tuple[int, int]:
        """
        Calculates optimal (grid_w, grid_h) maximizing resolution while respecting max_slices.
        """
        aspect_ratio = original_width / original_height
        candidate_grids = [
            (1, 1), (1, 2), (2, 1), (1, 3), (3, 1),
            (2, 2), (1, 4), (4, 1), (1, 5), (5, 1)
        ]

        best_grid = (1, 1)
        min_diff = float("inf")

        for gw, gh in candidate_grids:
            if gw * gh <= self.max_slices:
                grid_ratio = gw / gh
                diff = abs(aspect_ratio - grid_ratio)
                if diff < min_diff:
                    min_diff = diff
                    best_grid = (gw, gh)

        return best_grid

    def slice_image(self, pil_image: Image.Image) -> torch.Tensor:
        """
        Slices PIL image into a batch of (1 + N) tiles of size [tile_size, tile_size].
        Returns:
            tiles: Tensor of shape [1 + N, 3, tile_size, tile_size]
        """
        w, h = pil_image.size
        grid_w, grid_h = self.find_best_grid(w, h)

        # 1. Global overview tile (resized to tile_size x tile_size)
        global_tile = pil_image.resize((self.tile_size, self.tile_size), Image.Resampling.BICUBIC)
        global_tensor = self.transform(global_tile).unsqueeze(0)

        # 2. Local detailed tiles
        target_w = grid_w * self.tile_size
        target_h = grid_h * self.tile_size
        resized_img = pil_image.resize((target_w, target_h), Image.Resampling.BICUBIC)
        img_tensor = self.transform(resized_img) # [3, target_h, target_w]

        # Extract non-overlapping tiles
        local_tiles = []
        for i in range(grid_h):
            for j in range(grid_w):
                box_y = i * self.tile_size
                box_x = j * self.tile_size
                tile = img_tensor[:, box_y:box_y + self.tile_size, box_x:box_x + self.tile_size]
                local_tiles.append(tile.unsqueeze(0))

        # Concatenate global tile and local detail tiles
        all_tiles = torch.cat([global_tensor] + local_tiles, dim=0) # [1 + N, 3, 336, 336]
        return all_tiles

Step 2: Spatiotemporal Video Token Pooler

To compress 4,000 video frames into long-context sequences, the spatiotemporal pooler aggregates temporal redundancy across frame blocks:

class SpatiotemporalVideoPooler(nn.Module):
    """
    Hierarchically compresses dense video frame representations across time.
    Reduces 4,000 video frames into a compact multimodal token stream.
    """
    def __init__(self, token_dim: int = 4096, temporal_stride: int = 4):
        super().__init__()
        self.temporal_stride = temporal_stride
        self.pool = nn.AvgPool1d(kernel_size=temporal_stride, stride=temporal_stride)
        self.proj = nn.Linear(token_dim, token_dim)
        self.norm = nn.RMSNorm(token_dim)

    def forward(self, frame_tokens: torch.Tensor) -> torch.Tensor:
        """
        frame_tokens: [Batch, TotalFrames, TokensPerFrame, Dim]
        Returns:
            compressed: [Batch, CompressedTokens, Dim]
        """
        b, num_frames, tokens_per_frame, dim = frame_tokens.shape

        # Permute to compress across temporal frame axis: [B * TokensPerFrame, Dim, NumFrames]
        x = frame_tokens.permute(0, 2, 3, 1).reshape(b * tokens_per_frame, dim, num_frames)
        x_pooled = self.pool(x) # [B * TokensPerFrame, Dim, NumFrames // stride]

        # Reshape back to sequence form: [B, (NumFrames // stride) * TokensPerFrame, Dim]
        new_frames = x_pooled.shape[-1]
        x_out = x_pooled.reshape(b, tokens_per_frame, dim, new_frames).permute(0, 3, 1, 2)
        x_seq = x_out.reshape(b, new_frames * tokens_per_frame, dim)

        return self.norm(self.proj(x_seq))

Step 3: Production Pipeline: Multi-Image and Video Inference

When analyzing multi-page PDF documents, financial filings, or video streams scraped via our web scraping platform or real-time data fetching service, deploying InternLM-XComposer2.5 requires only a few lines of code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

class XComposerInferenceService:
    """
    High-capacity multimodal service for multi-image and long-video reasoning.
    Supports up to 96 interleaved images and 96k context windows.
    """
    def __init__(self, model_name: str = "internlm/internlm-xcomposer2d5-7b", device: str = "cuda"):
        self.device = device if torch.cuda.is_available() else "cpu"
        print(f"Loading InternLM-XComposer2.5 from [{model_name}] on {self.device}...")

        self.tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.bfloat16,
            device_map=self.device,
            trust_remote_code=True
        ).eval()
        print("XComposer2.5 initialized! Multi-image & video reasoning ready.")

    @torch.no_grad()
    def analyze_document_collection(self, image_paths: List[str], query: str) -> str:
        """
        Processes up to 96 high-resolution images in a single conversational prompt.
        """
        # Load and prepare image list
        images = [Image.open(p).convert("RGB") for p in image_paths]

        # Build multi-image prompt formatting
        image_tags = "".join([f"Image {idx+1}: <ImageHere> " for idx in range(len(images))])
        prompt = f"{image_tags}\nQuestion: {query}\nAnswer:"

        response, _ = self.model.chat(
            self.tokenizer,
            query=prompt,
            image=images,
            max_new_tokens=1024,
            do_sample=False
        )
        return response

# Usage Example:
# service = XComposerInferenceService()
# report = service.analyze_document_collection(
#     ["page1.png", "page2.png", "page3.png", "page4.png"],
#     "Cross-reference the financial tables on pages 2 and 4. What was the net margin change?"
# )

Production Deployment & Inference Optimization Best Practices

Deploying 96k-context multimodal models in production microservices requires disciplined memory management:

1. Dynamic Resolution Budgeting (Token Caps)

While the model supports up to 96 images, allowing each image to allocate 5 slices will flood the context window ($96 \times 5 \times 256 = 122,880 \text{ tokens}$):

Set a global visual token cap (e.g., 32,000 visual tokens).
If processing more than 20 images, configure max_slices = 2 or max_slices = 1 for background pages while reserving higher slice allocations for high-density tabular pages.

2. KV Cache Paging & Chunked Prefill

Processing a 96-image prompt requires an extensive prefill stage:

Deploy the model with PagedAttention (vLLM or SGLang) to eliminate GPU memory fragmentation.
Enable Chunked Prefill (chunk size: 8,192 tokens) to interleave prompt ingestion with active generation requests, preventing API timeouts.

3. Video Frame Uniform Sampling with Temporal Anchors

When analyzing 1-hour video files:

Sample frames at 1 FPS (3,600 frames total).
Insert explicit timestamp tokens (<timestamp_01:24>) before each frame’s visual token sequence. This anchors the LLM’s cross-attention to physical clock time, allowing exact temporal event localization.

Developers looking for detailed API endpoints, container configurations, and production SDKs can explore our extensive developer documentation.

Benchmark Evaluation: Multi-Image & Video Dominance

To validate InternLM-XComposer2.5’s capabilities, researchers tested the model against frontier proprietary and open-weights systems across multi-image and video benchmarks.

+-----------------------------------------------------------------------------------+
|                        Video-MME Benchmark (Overall Score)                        |
|                                                                                   |
|  InternLM-XComposer2.5 (7B) ====================================> 75.4% (SOTA)    |
|  Gemini 1.5 Pro (Proprietary)==================================> 75.0%             |
|  GPT-4V (Proprietary)       ==============================> 73.8%                 |
|  Qwen-VL-Max                ============================> 71.2%                   |
|  LLaVA-NeXT-Video (7B)      ===================> 52.3%                            |
+-----------------------------------------------------------------------------------+

Detailed Multimodal Benchmark Breakdown

The benchmark below compares zero-shot accuracy across multi-image document reasoning, mathematical vision, and video tasks:

Evaluation Benchmark	Domain / Task	LLaVA-NeXT-7B	GPT-4V	Gemini 1.5 Pro	InternLM-XComposer2.5 (7B)
Video-MME (Overall)	Video Comprehension	52.3%	73.8%	75.0%	75.4% (Top SOTA)
Video-MME (Long >30m)	Hour-Long Video QA	41.2%	67.5%	71.2%	72.1% (Top SOTA)
DocVQA	Multi-Page Documents	78.4%	88.4%	93.1%	94.8%
MathVista	Mathematical Vision	54.2%	67.7%	63.8%	68.2%
InfoVQA	Infographic Reasoning	58.6%	76.2%	78.4%	81.4%

Qualitative Analysis: Where XComposer2.5 Excels

Multi-Page Financial Cross-Referencing: In multi-page annual reports, the model cross-references cash flow line items on page 12 with balance sheet disclosures on page 48 without hallucination.
Dense Video Temporal Search: When queried “At what timestamp does the presenter switch from slide 3 to the live code terminal?”, the model returns the exact second mark across a 40-minute recording.
Complex Diagram Flow Tracking: In sprawling engineering diagrams and architectural blueprints, dynamic visual slicing preserves small connector arrows and component labels.

Conclusion: The Era of True Long-Context Multimodality

InternLM-XComposer2.5 demonstrates that multimodal foundation models do not need to choose between single-image resolution or long-context capacity. By integrating an extended 96k context window with aspect-ratio aware dynamic slicing and hierarchical temporal pooling, Shanghai AI Lab has created a compact 7B model that matches and exceeds proprietary frontier giants on multi-image and video reasoning.

As artificial intelligence moves toward autonomous desktop copilots, automated video auditing, and large-scale document intelligence, InternLM-XComposer2.5 provides a powerful, open-weights foundation for next-generation visual systems. To explore how your organization can integrate state-of-the-art vision and multimodal models into automated workflows, visit the AI Fetchly homepage.