feat(vision): encoder-free Gemma 4 image input (CPU + CUDA) (#250)

[pekkah]

Summary

Adds image input for Gemma 4 12B via the encoder-free "gemma4uv" projector — raw image patches projected straight into the LLM embedding space, no SigLIP ViT. Closes #250 (CPU image→text scope; GPU full-offload also included).

This is image understanding only (image in → text out). It is unrelated to the existing sharpi-cli image text→image generation (Z-Image-Turbo, SharpInference.Diffusion).

What's included

New SharpInference.Vision project

• VisionModel — loads/validates the gemma4uv mmproj GGUF
• GemmaUvVisionEmbedder — im2col(48) → LayerNorm×3 → linear → RMSNorm → bf16 input_projection; parity vs a numpy oracle (cosine > 0.9995)
• ImageIO — minimal PNG decoder (no imaging dependency, AOT-safe)
• ImagePreprocessor — calc_size_preserved_ratio + align-corners bilinear, mirroring mtmd-image.cpp

Decoder seam (IForwardPass.ForwardEmbedding)

• Injects a precomputed embedding: skips the sqrt(d) embedding scale (raw embeddings arrive final, gemma4.cpp:182) and builds PLE from the padding token (id 0)
• Implemented on both ForwardPass (CPU) and CudaForwardPass (CUDA full offload); gated by SupportsEmbeddingInput (default-throw elsewhere)

CLI

• --image (repeatable) + --mmproj
• Reference each image with an <image> marker in -p (left-to-right), or omit markers to prepend them
• Prompt rendered through the model's own chat template; each <|image|> placeholder (258880) expanded with its soft tokens wrapped in <|image>(255999) … <image|>(258882)
• Works under -g 0 (CPU) and -g -1 (CUDA)

Tooling

• scripts/gemma4uv_ref.py numpy oracle (golden fixtures in tests/fixtures/gemma4uv/, .f32 gitignored)
• download-model.ps1 now pulls the mmproj alongside the main GGUF

Verification (4070 Ti)

Path	Result
CPU -g 0, red image	"Red", logit 26.08, clean `<turn
CUDA -g -1, red image	"Red", logit 25.87 (faithful to CPU), ~50 t/s prefill
CUDA multi-image (first <image> … second <image>)	"red and blue" — correct order

Notes / findings

• The decisive fix: Gemma 4's turn format is <|turn>/<turn|> with a <|image|> placeholder — not Gemma 3's <start_of_turn>. Early garbage output was this format bug, not missing bidirectional attention; causal attention is sufficient.
• The current llama.cpp build cannot run this brand-new gemma4 mmproj (crashes 0xC0000409), so the numpy projector oracle + observed correctness is the validation backbone.

Follow-ups (out of scope)

• ForwardEmbedding on partial-offload hybrids and the Vulkan backend
• Audio (gemma4ua)
• Server image endpoint (content-blocks → same splice)
• Docs: design-doc gemma4uv section (this PR's docs/ update pending)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces multimodal vision support for Gemma 4 models by implementing the encoder-free gemma4uv image projector. It adds a new SharpInference.Vision project containing a custom PNG decoder, an image preprocessor, and the GemmaUvVisionEmbedder to project images into soft-token embeddings, while updating the CLI and forward pass backends to support splicing these embeddings. The review feedback highlights several improvement opportunities, including addressing potential integer overflow and validating image dimensions in the PNG decoder, optimizing bilinear interpolation in the preprocessor, validating the input buffer length in the vision embedder, catching more specific file I/O exceptions, and expanding special token compatibility for other Gemma versions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Potential integer overflow and out-of-bounds read/crash when validating chunk length clen in the PNG decoder. If clen is negative or extremely large, dataOff + clen + 4 can overflow and bypass the bounds check. Using clen > len - pos - 12 is safe from overflow.

            if (clen < 0 || clen > len - pos - 12)
                throw new InvalidDataException($"Truncated or invalid PNG chunk '{type}' with length {clen}.");

[gemini-code-assist]

Malformed PNG files with 0, negative, or extremely large dimensions can cause division-by-zero, NaN propagation, and negative array allocations in downstream preprocessing, or out-of-memory denial of service. Adding a validation check for dimensions prevents these issues.

                case "IHDR":
                    w = ReadUInt32BE(png, dataOff);
                    h = ReadUInt32BE(png, dataOff + 4);
                    if (w <= 0 || h <= 0 || w > 16384 || h > 16384)
                        throw new InvalidDataException($"Invalid or unsupported PNG dimensions: {w}x{h}.");
                    bitDepth = png[dataOff + 8];
                    colorType = png[dataOff + 9];
                    interlace = png[dataOff + 12];
                    seenIhdr = true;
                    break;

[gemini-code-assist]

Avoid redundant multiplications in the inner loop of bilinear interpolation. y0 * srcWidth and y1 * srcWidth can be calculated once in the outer loop to improve performance.

            int y0Offset = y0 * srcWidth;
            int y1Offset = y1 * srcWidth;
            for (int x = 0; x < w; x++)
            {
                float px = x * xRatio;
                int x0 = Math.Min((int)px, srcWidth - 1);
                int x1 = Math.Min(x0 + 1, srcWidth - 1);
                float xf = px - x0;

                int i00 = (y0Offset + x0) * 3, i10 = (y0Offset + x1) * 3;
                int i01 = (y1Offset + x0) * 3, i11 = (y1Offset + x1) * 3;

[gemini-code-assist]

Validate chw.Length before accessing it. If the input span is smaller than 3 * height * width, it will throw an unhandled IndexOutOfRangeException.

        if (width % _patch != 0 || height % _patch != 0)
            throw new ArgumentException($"image dims ({width}x{height}) must be multiples of {_patch}.");
        if (chw.Length < 3 * height * width)
            throw new ArgumentException($"chw length ({chw.Length}) is too small for dims {width}x{height}.");

[gemini-code-assist]

Catch specific exceptions such as UnauthorizedAccessException and SecurityException in addition to IOException to ensure precise error handling and graceful application exits when permission issues occur.

            catch (Exception ex) when (ex is IOException or NotSupportedException or InvalidDataException or UnauthorizedAccessException or System.Security.SecurityException)

References

1. When performing file I/O operations, catch specific exceptions (such as IOException, UnauthorizedAccessException, SecurityException, and NotSupportedException) instead of a generic Exception to ensure precise error handling and graceful application exits.

[gemini-code-assist]

Support both Gemma 4's <|image|>/<image|> and Gemma 3's <img>/<end_of_image> special tokens to ensure compatibility across different Gemma multimodal model versions.

        int imgOpen = tok.SpecialTokens.TryGetValue("<|image|>", out var o) ? o :
                      tok.SpecialTokens.TryGetValue("<img>", out var o2) ? o2 : 255999;
        int imgClose = tok.SpecialTokens.TryGetValue("<image|>", out var c) ? c :
                       tok.SpecialTokens.TryGetValue("<end_of_image>", out var c2) ? c2 : 258882;