feat(ai): add model dropdown for Ollama and custom OpenAI-compatible providers by jens-koesling · Pull Request #9692 · marimo-team/marimo

jens-koesling · 2026-05-26T20:00:17Z

📝 Summary

When adding a custom AI model in the AI Models settings, users previously had to manually type the model name. This PR adds automatic model discovery for Ollama and custom OpenAI-compatible providers.

Changes

New useProviderModels hook that fetches available models via the Marimo backend
New backend endpoint GET /api/ai/models that proxies requests to OpenAI-compatible /v1/models endpoints
Model text input is replaced with a dropdown when models are available
Model selection resets when switching providers (only if selected from dropdown, not manually typed)
CORS issues avoided by proxying through Marimo backend

Testing

Tested with Ollama and custom OpenAI-compatible providers.

Closes #5975

📋 Pre-Review Checklist

For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
Video or media evidence is provided for any visual changes (optional).

✅ Merge Checklist

I have read the contributor guidelines.
Documentation has been updated where applicable, including docstrings for API changes.
Tests have been added for the changes made.

Note: Tests for the backend endpoint and frontend hook will be added in a follow-up PR.

…providers - Add useProviderModels hook that fetches available models from /api/ai/models - Add backend endpoint GET /api/ai/models that proxies requests to OpenAI-compatible providers - Replace manual model text input with dropdown when models are available - Reset model selection when switching providers (only if model was selected from dropdown) - Avoids CORS issues by proxying through Marimo backend

vercel · 2026-05-26T20:00:24Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
marimo-docs	Ready	Preview, Comment	May 28, 2026 6:27pm

github-actions · 2026-05-26T20:00:31Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

jens-koesling · 2026-05-26T20:00:41Z

I have read the CLA Document and I hereby sign the CLA

for more information, see https://pre-commit.ci

dmadisetti

Thanks! I know ollama is a common option, but should be directly compatible with OpenAI api style inference. CC @Light2Dark for whether we want to support this. Hold on before hacking on this any more, but you'll also need to regen api schema (make codegen)

dmadisetti · 2026-05-26T22:41:29Z

+        LOGGER.warning(
+            "Could not fetch models from %s: %s", models_url, str(e)
+        )
+        return StructResponse({"models": []})


Do we just return nothing if we timeout?

Good point on the timeout handling. I chose to return an empty list on error/timeout so the UI gracefully falls back to manual text input. with this way the settings dialog still works even if the local provider is temporarily unavailable. Happy to change this to return a proper HTTP error code if preferred.

Light2Dark · 2026-05-28T05:34:45Z

Thanks for this @jens-koesling, I think another option was to call openai.clients list method.
Any thoughts on why you chose calling the API over the sdk?

import openai

oai_url = f'{llm_url}/v1'

oac = openai.Client(api_key=apikey, base_url=oai_url)
for oam in oac.models.list():
    print(oam.id)

for more information, see https://pre-commit.ci

jens-koesling · 2026-05-28T18:41:37Z

Thanks for this @jens-koesling, I think another option was to call openai.clients list method. Any thoughts on why you chose calling the API over the sdk?
import openai

oai_url = f'{llm_url}/v1'

oac = openai.Client(api_key=apikey, base_url=oai_url)
for oam in oac.models.list():
    print(oam.id)

Thanks for reviewing! Regarding the httpx vs SDK approach:
I chose the direct HTTP approach because I was considering to support other local model providers beyond Ollama (specifically LM Studio and similar OpenAI-compatible servers).
furthermore it is core dependency for Marimo while openai is optional.

My question: does using the OpenAI SDK require openai to be installed as a dependency on the client side? Or is the import openai in your example importing it via the Python environment that's already available in Marimo?

however the main advantage is that this approach works with any OpenAI-compatible endpoint (Ollama, LM Studio, vLLM, etc.) without requiring additional dependencies.
I am only running local models and don't have openai installed, so I was not even considering I could use this ;)

if you prefer the SDK approach I can change request method, no problem :)

Copilot

Pull request overview

This PR adds automatic AI model discovery for Ollama and custom OpenAI-compatible providers by introducing a backend proxy endpoint and a frontend hook, then updating the AI model settings UI to show a dropdown when models are available.

Changes:

Add GET /api/ai/models backend endpoint to fetch /v1/models from an OpenAI-compatible base_url.
Add useProviderModels hook to fetch provider models via the backend (CORS-safe).
Update AI settings “Add model” form to use a dropdown when model options are available and reset selection appropriately on provider changes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
packages/openapi/src/api.ts	Adds the typed OpenAPI path definition for `GET /api/ai/models`.
packages/openapi/api.yaml	Adds the OpenAPI spec entry for `GET /api/ai/models` with `base_url` query param.
marimo/_server/api/endpoints/ai.py	Implements the new `/api/ai/models` proxy endpoint.
frontend/src/components/app-config/use-provider-models.ts	Introduces a hook to fetch model lists from the backend endpoint.
frontend/src/components/app-config/ai-config.tsx	Switches the model input to a dropdown when provider models are available and wires in the new hook.

+    try:
+        async with httpx.AsyncClient(timeout=5.0) as client:
+            response = await client.get(models_url)
+            response.raise_for_status()
+            data = response.json()
+            model_ids = sorted(
+                [m["id"] for m in data.get("data", []) if m.get("id")]
+            )
+            return StructResponse(ProviderModelsResponse(models=model_ids))
+    except Exception as e:
+        LOGGER.warning(
+            "Could not fetch models from %s: %s", models_url, str(e)
+        )
+        return StructResponse(ProviderModelsResponse(models=[]))


+    base_url = request.query_params.get("base_url", "").rstrip("/")
+    if not base_url:
+        raise HTTPException(
+            status_code=HTTPStatus.BAD_REQUEST,
+            detail="base_url query parameter is required",
+        )
+
+    # Normalize URL
+    if base_url.endswith("/v1"):
+        models_url = f"{base_url}/models"
+    else:
+        models_url = f"{base_url}/v1/models"
+
+    try:
+        async with httpx.AsyncClient(timeout=5.0) as client:
+            response = await client.get(models_url)
+            response.raise_for_status()
+            data = response.json()
+            model_ids = sorted(
+                [m["id"] for m in data.get("data", []) if m.get("id")]
+            )
+            return StructResponse(ProviderModelsResponse(models=model_ids))


+@router.get("/models")
+@requires("edit")
+async def get_provider_models(
+    *,
+    request: Request,
+) -> Response:
+    """
+    parameters:
+        - in: query
+          name: base_url
+          schema:
+            type: string
+          required: true
+          description: The base URL of the OpenAI-compatible provider
+    responses:
+        200:
+            description: List of available models from the provider
+            content:
+                application/json:
+                    schema:
+                        type: object
+                        properties:
+                            models:
+                                type: array
+                                items:
+                                    type: string
+    """
+
+    base_url = request.query_params.get("base_url", "").rstrip("/")
+    if not base_url:
+        raise HTTPException(
+            status_code=HTTPStatus.BAD_REQUEST,
+            detail="base_url query parameter is required",
+        )
+


+      } finally {
+        setIsLoading(false);
+      }


  const providerName = isCustomProvider ? customProviderName : provider;
  const hasValidValues = providerName?.trim() && modelName?.trim();

+  // Resolve base URL für den gewählten Provider



+  // Resolve base URL für den gewählten Provider
+  const resolvedProviderName = isCustomProvider ? customProviderName : provider;
+  const ollamaBaseUrl = form.getValues("ai.ollama.base_url");


  const providerName = isCustomProvider ? customProviderName : provider;
  const hasValidValues = providerName?.trim() && modelName?.trim();

+  // Resolve base URL für den gewählten Provider



+  // Resolve base URL für den gewählten Provider
+  const resolvedProviderName = isCustomProvider ? customProviderName : provider;
+  const ollamaBaseUrl = form.getValues("ai.ollama.base_url");


Light2Dark

Sorry for the many comments from copilot & me. I can also have a look at implementing the changes if you'd like.

Id also add a comment on the frontend detailing why we do this in brief.

Light2Dark · 2026-05-29T05:35:57Z

 from typing import TYPE_CHECKING, Literal

+import httpx
+from pydantic import BaseModel


Users may not have pydantic. We should not require this as an import, only as TYPE_CHECKING import if needed.

Light2Dark · 2026-05-29T05:50:17Z

+  useEffect(() => {
+    if (!baseUrl) {
+      setModels([]);
+      setError(null);
+      return;
+    }
+
+    const controller = new AbortController();
+
+    const fetchModels = async () => {
+      setIsLoading(true);
+      setError(null);


consider using useAsyncData, it's a similar pattern

[pre-commit.ci] auto fixes from pre-commit.com hooks

b66d9dc

for more information, see https://pre-commit.ci

vercel Bot deployed to Preview May 26, 2026 20:02 View deployment

dmadisetti requested a review from Light2Dark May 26, 2026 22:42

dmadisetti reviewed May 26, 2026

View reviewed changes

dmadisetti added the enhancement New feature or request label May 27, 2026

chore: regenerate API schema after adding /api/ai/models endpoint

e9be54a

github-actions Bot added the bash-focus Area to focus on during release bug bash label May 27, 2026

vercel Bot deployed to Preview May 27, 2026 19:51 View deployment

jens-koesling and others added 2 commits May 28, 2026 20:23

fix: use Pydantic model for ProviderModelsResponse and fix import order

b08d61e

[pre-commit.ci] auto fixes from pre-commit.com hooks

51d2e62

for more information, see https://pre-commit.ci

vercel Bot deployed to Preview May 28, 2026 18:27 View deployment

Light2Dark requested a review from Copilot May 29, 2026 05:22

Copilot started reviewing on behalf of Light2Dark May 29, 2026 05:22 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Light2Dark reviewed May 29, 2026

View reviewed changes

Conversation

jens-koesling commented May 26, 2026

📝 Summary

Changes

Testing

📋 Pre-Review Checklist

✅ Merge Checklist

Uh oh!

vercel Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jens-koesling commented May 26, 2026

Uh oh!

dmadisetti left a comment

Choose a reason for hiding this comment

Uh oh!

dmadisetti May 26, 2026

Choose a reason for hiding this comment

Uh oh!

jens-koesling May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Light2Dark commented May 28, 2026

Uh oh!

jens-koesling commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Light2Dark left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Light2Dark May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Light2Dark May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vercel Bot commented May 26, 2026 •

edited

Loading

github-actions Bot commented May 26, 2026 •

edited

Loading

jens-koesling May 27, 2026 •

edited

Loading

jens-koesling commented May 28, 2026 •

edited

Loading

Light2Dark left a comment •

edited

Loading