Gemma 4 12B encoder-free multimodal (vision/audio) input support

[pekkah]

Summary

Add multimodal input support (image first, audio/video later) for Gemma 4 12B, which is a unified, encoder-free multimodal model. Today sharpi loads only the text decoder of gemma-4-12b-it-qat-q4_0.gguf and has no image/audio input path.

Background

Gemma 4 12B (released 2026-06-03) drops the separate SigLIP-style vision encoder used by earlier Gemma. Instead it projects raw image patches and audio waveforms directly into the LLM embedding space through lightweight linear layers (encoder-free / "unified"). See:

• https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/
• https://huggingface.co/blog/gemma4
• https://ai.google.dev/gemma/docs/core

GGUF packaging

Even though it's encoder-free, in the GGUF/llama.cpp ecosystem the projection tensors still ship as a small companion mmproj (~175 MB, vs the multi-hundred-MB SigLIP encoders of the encoder-based Gemma 4 sizes). The official repo google/gemma-4-12B-it-qat-q4_0-gguf contains exactly two files:

File	Size	Role
gemma-4-12b-it-qat-q4_0.gguf	6,975,877,728 B	Text LLM (decoder) — what we load today
mmproj-gemma-4-12b-it-qat-q4_0.gguf	175,115,264 B	Multimodal projector (vision + audio)

Inspecting the text GGUF confirms it carries only text metadata (gemma4.attention.*, gemma4.rope.*, gemma4.embedding_length_per_layer_input for PLE) and no vision/audio tensors — those live in the mmproj.

What's done

• scripts/download-model.ps1: the gemma4-12b-qat entry now pulls the mmproj alongside the text GGUF, and the disk-space guard counts only missing files (so the 175 MB top-up isn't blocked by the 7 GB bundle size).

Scope (engine work — not yet done)

• Parse the mmproj GGUF (projector tensors + config).
• Implement the encoder-free patch embedder: image preprocessing (variable aspect-ratio / resolution tiling) → patch projection → soft-token insertion into the embedding sequence.
• Wire image input through the CLI and the API server (OpenAI/Anthropic image-content parts).
• Forward-pass interleaving of image soft-tokens with text tokens (Gemma 4 supports freely interleaved multimodal input).
• (Later) audio waveform projection; video = frame sequence.
• Tests: projector parity against llama.cpp libmtmd on a fixed image.

Related

• Add Gemma 4 12B (dense gemma4_unified) support — CUDA-Hybrid/CUDA first, Q4_K_M #124 — Gemma 4 12B bring-up (text path)
• docs/gemma4-e4b-vision-plan.md — existing vision plan
• The gemma4-e4b-qat model also ships an mmproj we currently skip; same work would cover it.

[pekkah]

Shipped via PR #251 (squash-merged to master as 0a1ff40).

Done — Gemma 4 12B encoder-free (gemma4uv) IMAGE input, CPU + CUDA:

• SharpInference.Vision project (mmproj loader, projector forward w/ numpy-oracle parity cosine > 0.9995, PNG decoder, preprocessing)
• IForwardPass.ForwardEmbedding splice on CPU (ForwardPass) and CUDA full-offload (CudaForwardPass)
• CLI --image (repeatable) + <image> markers + --mmproj
• Verified: red→"Red" (CPU 26.08 / CUDA 25.87), multi-image→"red and blue" in order
• Docs: docs/SharpInference-Design.md §8.3

Deferred follow-ups (not in this PR):

• Audio input (the 12B mmproj also ships an audio projector) — not implemented; this issue's title mentions audio but only image landed.
• ForwardEmbedding on partial-offload hybrids and the Vulkan backend.
• Server image-content-block endpoint (/v1/messages, /v1/chat/completions).

E4B vision is a separate, different architecture (gemma4v ViT + gemma4a audio, not gemma4uv) tracked in #126. If 12B audio should be tracked, a dedicated follow-up issue is the cleanest home.