feat(models): port Granite 4 Vision (window-QFormer + Granite-4 hybrid)

docs/supported-models.md

@@ -83,6 +83,7 @@ Implemented VLM variants include:
 - GLM-4V (`glm4v`): GLM-4V ViT vision tower (3D patch embedding, bilinear-resampled learned position embeddings, Conv2d spatial downsample, SwiGLU patch merger) plus a GLM-4 text backbone driven by sectioned even/odd MRoPE. Reuses the shared Qwen-VL image processor and prompt/token plumbing.
 - GLM-4V MoE (`glm4v_moe`): GLM-4.5V-class variant reusing the GLM-4V ViT vision tower with a GLM-4 MoE text backbone (grouped `noaux_tc` routing, shared experts, `first_k_dense_replace` dense layers) driven by sectioned half-split MRoPE. Reuses the GLM-4 MoE machinery and the shared Qwen-VL runtime.
 - Granite Vision (`granite_vision`, or `llava_next` with a `granite` text config): IBM's document VLM. A SigLIP vision tower with four intermediate feature taps (concatenated on the channel axis) feeds a 2-layer GELU projector; a learned `image_newline` embedding, LLaVA-Next AnyRes multi-tile preprocessing (`image_grid_pinpoints`), and a dense Granite text backbone (embedding / attention / residual / logits multipliers) complete the model. Both `config.json` spellings route to the same loader. Best for document understanding, charts, and tables.
+- Granite 4 Vision (`granite4_vision`): IBM's document VLM with multi-depth visual injection. A SigLIP tower feeds eight window-QFormer projectors (a QFormer with self+cross attention per `4x4`/`8x8` window, with deepstack mean-pool and spatial strided-offset query downsamplers) whose packed outputs are added into the residual stream at eight different depths of a Granite 4 hybrid (`granitemoehybrid`) text backbone during prefill, rather than merged once. Reuses the shared AnyRes tiling and the four Granite scalar multipliers. Best for document understanding.
 - GLM-OCR (`glm_ocr`): document-OCR sibling of GLM-4V. A 24-block ViT (3D patch embedding, per-head q/k RMSNorm on the packed `cu_seqlens` attention, 2D vision RoPE, Conv2d spatial downsample, SwiGLU patch merger) feeds a 16-layer GLM-4 text decoder driven by full-width even/odd MRoPE (`rope_parameters` with `mrope_section [16, 24, 24]`, `partial_rotary_factor 1.0`). The tower has no learned position embedding or post-conv norm, and the loader drops the next-n prediction (MTP) layer. Patches are reordered from the processor's raster order into spatial-merge-window order so the rotary, downsample, and merged-token grid stay spatially aligned (OCR reads scrambled patches wrong). Best for plain text, tables, and formula recognition.
 - Youtu-VL
 - Kimi-VL and Kimi-VL 2.5 (`kimi_vl` / `kimi_k25`): MoonViT native-resolution vision encoder (Conv2d patch embedding, learned plus bicubically-interpolated 2D position embedding, a shared 2D rotary embedding, block-diagonal cross-image attention, and `spatial_merge_size` patch merging) feeding a `LayerNorm -> Linear -> GELU -> Linear` connector into a DeepSeek-V3-style MoE text backbone. Detected, loaded, and served end to end: the safetensors directory loader wires the MoonViT tower, the connector, and the DeepSeek-V3 MoE backbone; the native-resolution processor patchifies each image; and the runtime expands each `<|media_pad|>` placeholder into `(h/merge) * (w/merge)` tokens before the merged vision features replace them. Image path only; the Kimi-VL 2.5 3D MoonViT video (image plus video patch embedding) path is tracked as a separate follow-up.

src/commands/generate_vlm.rs

@@ -218,6 +218,15 @@ fn print_preparation_summary(summary: VlmPreparationSummary) {
                 image_blocks, total_image_tokens
             );
         }
+        VlmPreparationSummary::Granite4Vision {
+            image_blocks,
+            total_image_tokens,
+        } => {
+            println!(
+                "Granite 4 Vision: expanded {} <image> placeholder(s) ({} total image tokens)",
+                image_blocks, total_image_tokens
+            );
+        }
         VlmPreparationSummary::ImageBlocks(stats) => match stats.action {
             ImageTokenBlockAction::Expanded {
                 existing_image_count,

src/distributed/tensor_parallel/inference.rs

@@ -258,6 +258,9 @@ fn fallback_architecture(model_type: ModelType) -> &'static str {
         // Granite Vision's text backbone is Granite; TP is refused for VLM-kind
         // models earlier, this keeps the dispatch table total.
         ModelType::GraniteVisionVLM => "granite",
+        // Granite 4 Vision's text backbone is granitemoehybrid; TP is refused for
+        // VLM-kind models earlier, this keeps the dispatch table total.
+        ModelType::Granite4VisionVLM => "granitemoehybrid",
         // Youtu-VL is not currently supported by tensor-parallel inference;
         // we return a placeholder architecture string here so the planner
         // does not panic on the dispatch table lookup. The actual loader

src/loaded_model.rs

@@ -78,6 +78,7 @@ pub enum LoadedModel {
     MllamaVLM(vision::MllamaVLModel),
     LlavaVLM(vision::VisionLanguageModel),
     GraniteVisionVLM(vision::GraniteVisionVLModel),
+    Granite4VisionVLM(vision::Granite4VisionVLModel),
     Qwen2VL(vision::Qwen2VLModel),
     Qwen25VL(vision::Qwen25VLModel),
     Qwen3VL(vision::Qwen3VLModel),
@@ -210,6 +211,7 @@ macro_rules! delegate_language_model {
             LoadedModel::MllamaVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::LlavaVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::GraniteVisionVLM(inner) => LanguageModel::$method(inner, $($arg),*),
+            LoadedModel::Granite4VisionVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Qwen2VL(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Qwen25VL(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Qwen3VL(inner) => LanguageModel::$method(inner, $($arg),*),

src/loaded_model_capabilities.rs

@@ -61,6 +61,7 @@ pub enum VlmRuntimeRef<'a> {
     /// LFM2-VL (lfm2_vl) runtime.
     Lfm2Vl(&'a vision::Lfm2VlModel),
     GraniteVision(&'a vision::GraniteVisionVLModel),
+    Granite4Vision(&'a vision::Granite4VisionVLModel),
     Standard(&'a vision::VisionModule),
 }
 
@@ -175,6 +176,7 @@ impl LoadedModel {
             Self::Llama4VLM(vlm) => Some(VlmRuntimeRef::Standard(&vlm.vision)),
             Self::LlavaVLM(vlm) => Some(VlmRuntimeRef::Standard(&vlm.vision)),
             Self::GraniteVisionVLM(model) => Some(VlmRuntimeRef::GraniteVision(model)),
+            Self::Granite4VisionVLM(model) => Some(VlmRuntimeRef::Granite4Vision(model)),
             _ => None,
         }
     }

src/loading/mod.rs

@@ -176,6 +176,7 @@ fn try_load_vlm_model_from_dir(
         ModelType::Gemma4Unified => Some(load_gemma4_unified(model_path)?),
         ModelType::LlavaVLM => Some(load_llava_vlm(model_path)?),
         ModelType::GraniteVisionVLM => Some(load_granite_vision_vlm(model_path)?),
+        ModelType::Granite4VisionVLM => Some(load_granite4_vision_vlm(model_path)?),
         ModelType::LlavaBunnyVLM => Some(load_llava_bunny_vlm(model_path)?),
         ModelType::AyaVisionVLM => Some(load_aya_vision_vlm(model_path)?),
         ModelType::PaliGemmaVLM => Some(load_paligemma_vlm(model_path)?),

src/loading/vlm.rs

@@ -41,6 +41,8 @@ use models::sanitize_config_json;
 mod gemma;
 #[path = "vlm_gemma_unified.rs"]
 mod gemma_unified;
+#[path = "vlm_granite4_vision.rs"]
+mod granite4_vision;
 #[path = "vlm_granite_vision.rs"]
 mod granite_vision;
 #[path = "vlm_idefics2.rs"]
@@ -75,6 +77,7 @@ mod youtu_vl_loader;
 pub(crate) use gemma::{load_gemma3_vlm, load_gemma3n_vlm, load_gemma4_vlm};
 pub(crate) use gemma_unified::load_gemma4_unified;
 pub(crate) use granite_vision::load_granite_vision_vlm;
+pub(crate) use granite4_vision::load_granite4_vision_vlm;
 pub(crate) use idefics2::load_idefics2_vlm;
 pub(crate) use internvl::load_internvl_vlm;
 pub(crate) use kimi_vl_loader::load_kimi_vl_vlm;

src/loading/vlm_granite4_vision.rs

@@ -0,0 +1,254 @@
+// Copyright 2025-2026 Lablup Inc. and Jeongkyu Shin
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Granite 4 Vision loader (SigLIP + 8 window-QFormer projectors + Granite 4
+//! hybrid text backbone with multi-depth injection).
+
+use anyhow::Result;
+use std::path::Path;
+
+use crate::LoadedModel;
+use crate::models;
+use crate::vision;
+use crate::vision::connectors::granite4_vision::{Downsampler, WindowQFormerProjector};
+
+use super::{load_vlm_weights_common, read_sanitized_vlm_config, strip_language_model_prefix};
+
+/// Default LLaVA-Next `image_grid_pinpoints` for Granite 4 Vision.
+fn default_pinpoints() -> Vec<(i32, i32)> {
+    let mut p: Vec<(i32, i32)> = Vec::new();
+    for w in (384..=3840).step_by(384) {
+        p.push((384, w));
+    }
+    for w in (384..=1920).step_by(384) {
+        p.push((768, w));
+    }
+    for w in (384..=1152).step_by(384) {
+        p.push((1152, w));
+    }
+    for h in [1536, 1920] {
+        for w in [384, 768] {
+            p.push((h, w));
+        }
+    }
+    for h in [2304, 2688, 3072, 3456, 3840] {
+        p.push((h, 384));
+    }
+    p
+}
+
+fn parse_pinpoints(full_config: &serde_json::Value) -> Vec<(i32, i32)> {
+    full_config
+        .get("image_grid_pinpoints")
+        .and_then(|v| v.as_array())
+        .map(|arr| {
+            arr.iter()
+                .filter_map(|pair| {
+                    let p = pair.as_array()?;
+                    Some((p.first()?.as_i64()? as i32, p.get(1)?.as_i64()? as i32))
+                })
+                .collect::<Vec<_>>()
+        })
+        .filter(|v| !v.is_empty())
+        .unwrap_or_else(default_pinpoints)
+}
+
+/// `downsample_rate` `"q/w"` -> `(q, w)`; defaults to `(4, 8)`.
+fn parse_downsample_rate(full_config: &serde_json::Value) -> (i32, i32) {
+    full_config
+        .get("downsample_rate")
+        .and_then(|v| v.as_str())
+        .and_then(|s| {
+            let mut it = s.split('/');
+            let q = it.next()?.trim().parse::<i32>().ok()?;
+            let w = it.next()?.trim().parse::<i32>().ok()?;
+            Some((q, w))
+        })
+        .unwrap_or((4, 8))
+}
+
+fn parse_i32_pairs(v: Option<&serde_json::Value>) -> Vec<[i32; 2]> {
+    v.and_then(|v| v.as_array())
+        .map(|arr| {
+            arr.iter()
+                .filter_map(|pair| {
+                    let p = pair.as_array()?;
+                    Some([p.first()?.as_i64()? as i32, p.get(1)?.as_i64()? as i32])
+                })
+                .collect()
+        })
+        .unwrap_or_default()
+}
+
+fn parse_i32_list(v: Option<&serde_json::Value>) -> Vec<i32> {
+    v.and_then(|v| v.as_array())
+        .map(|arr| {
+            arr.iter()
+                .filter_map(|x| x.as_i64().map(|i| i as i32))
+                .collect()
+        })
+        .unwrap_or_default()
+}
+
+/// Load a Granite 4 Vision VLM.
+pub(crate) fn load_granite4_vision_vlm(model_path: &Path) -> Result<LoadedModel> {
+    let (_config_str, full_config) = read_sanitized_vlm_config(model_path)?;
+
+    // Text backbone config: inherit top-level quantization, parse as granitemoehybrid.
+    let mut text_config_val = full_config
+        .get("text_config")
+        .cloned()
+        .ok_or_else(|| anyhow::anyhow!("Missing text_config in config.json"))?;
+    if let Some(obj) = text_config_val.as_object_mut()
+        && obj.get("quantization").is_none()
+        && let Some(q) = full_config.get("quantization")
+    {
+        obj.insert("quantization".to_string(), q.clone());
+    }
+    let text_args: models::granitemoehybrid::ModelArgs = serde_json::from_value(text_config_val)
+        .map_err(|e| anyhow::anyhow!("Failed to parse Granite 4 text config: {}", e))?;
+
+    let vision_config: vision::config::VisionConfig = serde_json::from_value(
+        full_config
+            .get("vision_config")
+            .cloned()
+            .ok_or_else(|| anyhow::anyhow!("Missing vision_config in config.json"))?,
+    )
+    .map_err(|e| anyhow::anyhow!("Failed to parse Granite 4 vision config: {}", e))?;
+
+    let group_size = text_args.group_size();
+    let bits = text_args.bits();
+    let (q, w) = parse_downsample_rate(&full_config);
+
+    let weights = strip_language_model_prefix(load_vlm_weights_common(model_path, None)?);
+
+    // Vision tower and 8 window-QFormer projectors read by ref before the text
+    // backbone consumes the weight map. The granite4 SigLIP tower uses the
+    // sigmoid `GELU(approx="fast")` in its MLP (unlike Granite Vision #539, which
+    // uses the tanh variant), matching the reference vision encoder.
+    let vision_tower =
+        vision::encoders::siglip::SigLipVisionModel::from_weights_with_quant_and_gelu(
+            &weights,
+            &vision_config,
+            "vision_tower.vision_model",
+            group_size,
+            bits,
+            true, // use_fast_gelu (sigmoid GELU)
+        )
+        .map_err(|e| anyhow::anyhow!("Failed to load Granite 4 vision tower: {}", e))?;
+
+    let spatial_offsets = [(0, 0), (0, 1), (1, 0), (1, 1)];
+    let mut projectors = Vec::with_capacity(8);
+    for i in 0..4 {
+        projectors.push(
+            WindowQFormerProjector::from_weights(
+                &weights,
+                &format!("layerwise_projectors.{i}"),
+                Downsampler::MeanPool,
+                q,
+                w,
+                group_size,
+                bits,
+            )
+            .map_err(|e| anyhow::anyhow!("Failed to load layerwise projector {i}: {}", e))?,
+        );
+    }
+    for (i, (row_off, col_off)) in spatial_offsets.iter().enumerate() {
+        projectors.push(
+            WindowQFormerProjector::from_weights(
+                &weights,
+                &format!("spatial_projectors.{i}"),
+                Downsampler::Strided {
+                    row_off: *row_off,
+                    col_off: *col_off,
+                },
+                q,
+                w,
+                group_size,
+                bits,
+            )
+            .map_err(|e| anyhow::anyhow!("Failed to load spatial projector {i}: {}", e))?,
+        );
+    }
+
+    let image_newline = weights
+        .get("image_newline")
+        .map(|x| mlxcel_core::copy(x))
+        .ok_or_else(|| anyhow::anyhow!("Missing image_newline weight"))?;
+
+    // Stream routing from config.
+    let deepstack_map = parse_i32_pairs(full_config.get("deepstack_layer_map"));
+    let spatial_targets = parse_i32_list(full_config.get("spatial_target_layers"));
+    let spatial_vision_layer = full_config
+        .get("spatial_vision_layer")
+        .and_then(|v| v.as_i64())
+        .unwrap_or(-1) as i32;
+    let use_spatial = full_config
+        .get("use_spatial_sampling")
+        .and_then(|v| v.as_bool())
+        .unwrap_or(true);
+
+    // Collect the deepstack vision taps; the spatial tap reuses one of them when
+    // present, otherwise it is appended.
+    let mut taps: Vec<i32> = deepstack_map.iter().map(|p| p[0]).collect();
+    let spatial_tap_idx = match taps.iter().position(|&t| t == spatial_vision_layer) {
+        Some(idx) => idx,
+        None => {
+            taps.push(spatial_vision_layer);
+            taps.len() - 1
+        }
+    };
+
+    let mut stream_specs: Vec<(usize, usize, usize)> = Vec::new();
+    for (i, pair) in deepstack_map.iter().enumerate() {
+        stream_specs.push((i, i, pair[1] as usize));
+    }
+    if use_spatial {
+        for (i, &layer) in spatial_targets.iter().enumerate() {
+            stream_specs.push((4 + i, spatial_tap_idx, layer as usize));
+        }
+    }
+
+    let image_token_index = full_config
+        .get("image_token_index")
+        .and_then(|v| v.as_i64())
+        .unwrap_or(100352) as i32;
+    let pinpoints = parse_pinpoints(&full_config);
+    let image_size = vision_config.image_size as u32;
+    let patch_grid = (vision_config.image_size / vision_config.patch_size) as i32; // 24
+    let feature_side = patch_grid / (w / q); // 24 / 2 = 12
+    let base_tokens = feature_side * feature_side; // 144
+    let processor = vision::processors::anyres::AnyResProcessor::new(pinpoints, image_size);
+
+    // Text backbone consumes the weight map last.
+    let eos_token_id = text_args.eos_token_ids().first().copied().unwrap_or(100257);
+    let text_model = models::GraniteMoeHybridModel::from_weights(text_args, weights)
+        .map_err(|e| anyhow::anyhow!("Failed to load Granite 4 text model: {}", e))?;
+
+    let vlm = vision::Granite4VisionVLModel::new(
+        text_model,
+        vision_tower,
+        projectors,
+        image_newline,
+        processor,
+        image_token_index,
+        taps,
+        stream_specs,
+        feature_side,
+        base_tokens,
+        eos_token_id,
+    );
+
+    Ok(LoadedModel::Granite4VisionVLM(vlm))
+}

src/model_metadata.rs

@@ -115,6 +115,7 @@ macro_rules! for_each_model_registration {
             Gemma4Unified => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("Gemma4 Unified cannot be loaded with LoRA adapters yet") };
             LlavaVLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("LLaVA VLM cannot be loaded with LoRA adapters yet") };
             GraniteVisionVLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("Granite Vision VLM cannot be loaded with LoRA adapters yet") };
+            Granite4VisionVLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("Granite 4 Vision VLM cannot be loaded with LoRA adapters yet") };
             LlavaBunnyVLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("LLaVA VLM cannot be loaded with LoRA adapters yet") };
             AyaVisionVLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("Aya Vision VLM cannot be loaded with LoRA adapters yet") };
             PaliGemmaVLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("PaliGemma VLM cannot be loaded with LoRA adapters yet") };

src/models/detection.rs

@@ -238,6 +238,7 @@ pub fn get_model_type(model_path: &Path) -> Result<ModelType> {
         "moondream3" => Ok(ModelType::Moondream3VLM),
         "moondream2" | "moondream1" => Ok(ModelType::Moondream2VLM),
         "granite_vision" => Ok(ModelType::GraniteVisionVLM),
+        "granite4_vision" => Ok(ModelType::Granite4VisionVLM),
         "llava" | "llava_next" => {
             // The original IBM Granite Vision checkpoint ships as `llava_next`
             // with a `granite` text backbone; route it to the Granite VLM.