feat(server): multimodal image input endpoint (Gemma 4 gemma4uv) (#253)

[pekkah]

Closes #253.

Wires encoder-free Gemma 4 vision (the CLI's --image from #250) through the API
server's Anthropic (/v1/messages) and OpenAI (/v1/chat/completions) endpoints.

Design — engine owns vision, server passes bytes

• InferenceEngine.EnableImageInput(...), SupportsImageInput, and
  GenerateImageChunksAsync(...). SplicePrefillImages projects each image
  (decode → preprocess → embed) and splices open + soft tokens + close at every
  <|image|> placeholder during a fresh prefill (mirrors the CLI's RunImagePrompt).
• IInferenceEngine gains default interface members so existing implementers inherit
  a "no image support" stub.
• InferenceEngineLoader opens the mmproj VisionModel when configured and rejects
  unsupported configs (non-embedding pass, batched, or a non-Gemma text model) with a clear error.

Server surface

• New ImageContent helper: base64 / data-URL decode + full PNG decode-validation at parse
  time (unsupported / corrupt / decompression-bomb payloads → HTTP 400 before any response).
• Anthropic image content blocks (source.type=base64) and OpenAI image_url /
  input_image parts (data URL). Each image is collected and replaced inline by the
  model's <|image|> placeholder, multi-image per message in document order (capped at 16/request).
• MmprojPath option settable via SHARPI_MMPROJ env, the SharpInference:MmprojPath
  config key, or Configure(o => o.MmprojPath = ...).

Gemma 4 thinking default

Gemma 4 is not a reasoning model. The server defaults thinking off for it
(ChatTemplateRenderer.ModelDefaultsThinkingOff), matching the CLI and applied uniformly
across /v1/messages, /v1/chat/completions, and /v1/responses. An explicit request
flag still wins; SHARPI_NO_THINKING still forces off.

Review cycle

A multi-agent adversarial review of the initial diff found — and this PR fixes — two real
bugs that single-image smoke tests masked:

• Decode position: content tokens were forwarded at tokens.Length + i instead of
  startPos + i, so multi-token image responses decoded over the spliced soft-token KV with
  wrong RoPE positions (the repetition/garbling). Fixed → output is now clean and coherent.
• Cross-request KV reuse: the image path didn't invalidate _prevTokens, letting a later
  text request reuse image-soft-token KV. Fixed.
• Decompression-bomb DoS: ImageIO inflated the PNG IDAT unbounded; now bounded to the
  exact decoded size.
• Robustness: bad images / malformed JSON → clean 400 (not 500 / aborted stream); engine
  errors during generation are surfaced gracefully.

Tests / verification

• 9 wire-format image tests + MmprojPath configurability + ImageIO bomb-cap tests.
• Full solution builds clean in Release (TreatWarningsAsErrors); Tests.Server green
  (the 3 WithTools failures are a local-appsettings artifact, unrelated — they fail on
  master too).
• End-to-end on a 4070 Ti with the 12B gemma4uv model + its mmproj: both endpoints
  return clean, coherent descriptions of a real image (e.g. "A serene, misty landscape
  features a calm lake reflecting a mountain range under a clear blue sky.").

Deferred (noted, out of scope)

• /v1/responses does not yet accept image content (text-only; pre-existing).
• The Gemma 4 image token-id fallbacks are duplicated with the CLI (intentional, by the
  byte-identical-copy design).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request implements multimodal image input support (issue #253) for Gemma 4 models, allowing the engine to project images into soft-token blocks and splice them during prompt prefill. It updates the Anthropic and OpenAI endpoints to parse and validate base64-encoded PNGs or data URLs, routing them to the new image-aware generation path. The feedback focuses on enhancing robustness and defensive programming, specifically recommending validation checks on JSON elements during payload parsing in ExtractTextAndImages, ParseAnthropicImageBlock, and FlattenContent to prevent unhandled exceptions (HTTP 500 errors). Additionally, it is suggested to verify that image byte arrays are not null or empty before loading them into memory streams.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Robustness and Defensive Programming: The ExtractTextAndImages method iterates over el.EnumerateArray(). If the client sends a malformed JSON payload where the array contains non-object elements (e.g., strings or numbers), or if the type or text properties are not strings, calling block.TryGetProperty or t.GetString() will throw an unhandled InvalidOperationException. This will result in an HTTP 500 Internal Server Error instead of a clean HTTP 400 Bad Request. We should defensively check the ValueKind of the elements and properties before accessing them.

    private static string ExtractTextAndImages(JsonElement? contentEl, List<byte[]> images)
    {
        if (contentEl is null) return "";
        var el = contentEl.Value;
        if (el.ValueKind == JsonValueKind.String) return el.GetString() ?? "";
        if (el.ValueKind != JsonValueKind.Array) return "";

        var sb = new StringBuilder();
        foreach (var block in el.EnumerateArray())
        {
            if (block.ValueKind != JsonValueKind.Object)
                continue;
            var type = block.TryGetProperty("type", out var t) && t.ValueKind == JsonValueKind.String ? t.GetString() : null;
            if (type == "text")
            { 
                if (block.TryGetProperty("text", out var tv) && tv.ValueKind == JsonValueKind.String)
                    sb.Append(tv.GetString());
            }
            else if (type == "image")
            { 
                images.Add(ParseAnthropicImageBlock(block));
                sb.Append(ImageContent.Placeholder);
            }
        }
        return sb.ToString();
    }

[gemini-code-assist]

Robustness and Defensive Programming: In ParseAnthropicImageBlock, if the source property is not a JSON object, or if the type property is not a string, calling source.TryGetProperty or st.GetString() will throw an unhandled InvalidOperationException, leading to an HTTP 500 error. We should defensively check the ValueKind of source and st to ensure they are of the expected types, throwing an ImageContentException (which is cleanly caught and mapped to HTTP 400) if they are not.

    private static byte[] ParseAnthropicImageBlock(JsonElement block)
    {
        if (!block.TryGetProperty("source", out var source) || source.ValueKind != JsonValueKind.Object)
            throw new ImageContentException("image content block is missing a valid 'source' object.");
        var srcType = source.TryGetProperty("type", out var st) && st.ValueKind == JsonValueKind.String ? st.GetString() : null;
        if (srcType != "base64")
            throw new ImageContentException(
                $"unsupported image source type '{srcType}'; only base64 image sources are supported (URL fetch is not).");
        if (!source.TryGetProperty("data", out var data) || data.ValueKind != JsonValueKind.String)
            throw new ImageContentException("image source is missing base64 'data'.");
        return ImageContent.FromBase64(data.GetString() ?? "");
    }

[gemini-code-assist]

Robustness and Defensive Programming: In FlattenContent, if the client sends a malformed JSON payload where the array contains non-object elements, or if the type, text, or image_url properties are not of the expected JSON types, calling part.TryGetProperty, t.GetString(), or iu.TryGetProperty will throw an unhandled InvalidOperationException. This will result in an HTTP 500 Internal Server Error instead of a clean HTTP 400 Bad Request. We should defensively check the ValueKind of the elements and properties before accessing them.

    private static string FlattenContent(JsonElement? contentEl, List<byte[]> images)
    {
        if (contentEl is null) return "";
        var el = contentEl.Value;
        if (el.ValueKind == JsonValueKind.String) return el.GetString() ?? "";
        if (el.ValueKind != JsonValueKind.Array) return "";

        var sb = new StringBuilder();
        foreach (var part in el.EnumerateArray())
        {
            if (part.ValueKind != JsonValueKind.Object)
                continue;
            var type = part.TryGetProperty("type", out var t) && t.ValueKind == JsonValueKind.String ? t.GetString() : null;
            if (type == "text")
            { 
                if (part.TryGetProperty("text", out var tv) && tv.ValueKind == JsonValueKind.String)
                    sb.Append(tv.GetString());
            }
            else if (type is "image_url" or "input_image")
            { 
                // image_url is either {url:"data:..."} (chat completions) or a bare string url.
                string url = "";
                if (part.TryGetProperty("image_url", out var iu))
                {
                    if (iu.ValueKind == JsonValueKind.String)
                    {
                        url = iu.GetString() ?? "";
                    }
                    else if (iu.ValueKind == JsonValueKind.Object)
                    {
                        url = iu.TryGetProperty("url", out var u) && u.ValueKind == JsonValueKind.String ? u.GetString() ?? "" : "";
                    }
                }
                if (string.IsNullOrEmpty(url))
                    throw new ImageContentException($"{type} content part is missing a base64 'image_url'.");
                images.Add(ImageContent.FromDataUrl(url));
                sb.Append(ImageContent.Placeholder);
            }
        }
        return sb.ToString();
    }

[gemini-code-assist]

Defensive Programming: Ensure that each element in imageBytes is not null or empty before attempting to load it into a MemoryStream. If a null or empty byte array is passed, it will throw an unhandled ArgumentNullException or cause issues in ImageIO.LoadRgb.

        var blocks = new (float[] Soft, int NTok)[imageBytes.Count];
        for (int i = 0; i < imageBytes.Count; i++)
        {
            ct.ThrowIfCancellationRequested();
            if (imageBytes[i] is null || imageBytes[i].Length == 0)
                throw new ArgumentException($"Image at index {i} is null or empty.", nameof(imageBytes));
            byte[] rgb;
            int w, h;
            using (var ms = new MemoryStream(imageBytes[i], writable: false))