chat: harden peg-native tool call parsing by ServeurpersoCom · Pull Request #24329 · ggml-org/llama.cpp

ServeurpersoCom · 2026-06-09T07:39:38Z

Overview

While working we hit a silent bug on llama 3.3 dense: the assistant turn came back empty, no error shown. It only happened with tools enabled. The model emits a tool call in a variant format that the peg-native parser rejects, which blew up the whole turn. I landed a debug log first to confirm exactly what the model was emitting, then the actual fix: accept the "type": "function" variant, and fail soft on parse errors instead of tearing down the turn.

Additional information

A bug was discovered while working on this PR #23226 that allows the system to see when an error occurs; otherwise, it's silent (empty assistant turn). This PR adds the missing error if the PEG fails.

604615515-bd9a70de-baf7-44bb-99c4-9701cd714d17

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

accept an optional leading type: function field in build_json_tools_flat_keys so openai style tool calls parse on templates whose serialization opens on the name field. return a clean error and log the unparsed fragment on a final peg parse failure instead of throwing the raw parser position and input. keep the raw arguments string in func_args_not_string when it is not valid json instead of aborting the prompt render.

pwilkin · 2026-06-09T09:02:23Z

Yeah, I'd love to see the exact repro for this.

ServeurpersoCom · 2026-06-09T09:49:04Z

Yeah, I'd love to see the exact repro for this.

The problem look systematic with this model if tools are activated :
https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF/tree/main

I will minimize the PR to a simple error message if the parser fails.

a final peg parse failure threw the raw parser position and input. log the unparsed fragment and raise a clearer error instead, so a model output that does not match the expected format no longer fails silently with an empty assistant turn. minimal change, no behavior change on successful parses.

aldehir · 2026-06-09T15:03:26Z

FWIW I think the old parsing also handled this variation. I would like to see it's unconstrained output first because we may also need to adjust the grammar triggers.

ServeurpersoCom · 2026-06-09T15:32:20Z

FWIW I think the old parsing also handled this variation. I would like to see it's unconstrained output first because we may also need to adjust the grammar triggers.

I'll redo clean real captures from my server, but this model emits this:

get_weather:
{"type": "function", "name": "get_weather", "parameters": {"location": "Paris"}}

run_javascript:
{"type": "function", "name": "run_javascript", "parameters": {"code": "console.log(\"Salut !\");", "timeout_ms": 10000}}

EDIT, I confirm :

Clean captures, model unloaded after. Unconstrained output via /apply-template then raw /completion (no grammar, no parsing, temp 0) :

get_weather:
{"type": "function", "name": "get_weather", "parameters": {"location": "Paris"}}

run_javascript:
{"type": "function", "name": "run_javascript", "parameters": {"code": "console.log(\"Hello, world!\")", "timeout_ms": "1000"}}

two tools provided:
{"type": "function", "name": "get_weather", "parameters": {"location": "Paris"}}

It always opens with {"type": "function", ... and never the template's {"name": ..., so the lazy trigger on the name-field prefix never fires and generation stays unconstrained. The json itself is well formed, so this is a format mismatch (the type wrapper), not malformed output. Looks like the grammar triggers need to also engage on this opening.

ServeurpersoCom · 2026-06-09T17:06:42Z

So, preferred direction: parse tolerance for the leading "type": "function", a trigger adjustment so the grammar constrains generation back to {"name": ...}, or both? Happy to implement whichever you prefer.

aldehir · 2026-06-09T17:40:09Z

The parsing changes are fine but we should add {"type": "function", as an additional trigger. No need to overly constrain it.

ServeurpersoCom · 2026-06-09T18:44:41Z

ServeurpersoCom · 2026-06-09T18:53:24Z

I did a little non-regression test by having my different family of models draw Mandelbrot in ASCII art :) The only one that was broken was this llama 3.3 now fixed

aldehir · 2026-06-09T21:27:26Z

+        // Accept an optional leading "type": "function" field
+        auto type_field = optional(literal("\"type\"") + space() + literal(":") + space() +
+                                   literal("\"function\"") + space() + literal(",") + space());
+        auto ordered_body = tool_open(literal("{")) + space() + type_field;


Looks good! Let's just finish up by wrapping this parsing logic in the new flag you made from the analysis.

Done. Both the grammar trigger and the flat-keys parsing leniency are now gated on openai_wrapper_trigger, set only by the analysis workaround that matches Meta's JSON tool instruction. Other JSON_NATIVE templates no longer get the "type": "function" handling.

Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set.

aldehir · 2026-06-15T09:10:53Z

@pwilkin

pwilkin

Innocent enough :) would prob be better to have this in the main analysis route instead of a workaround (the criterion could be the same), but not a big deal.

ServeurpersoCom · 2026-06-15T13:36:17Z

Innocent enough :) would prob be better to have this in the main analysis route instead of a workaround (the criterion could be the same), but not a big deal.

Besides, it's an older model, not very efficient for toolcalling, but it's cleaner than a blank response.

* chat: harden peg-native tool call parsing accept an optional leading type: function field in build_json_tools_flat_keys so openai style tool calls parse on templates whose serialization opens on the name field. return a clean error and log the unparsed fragment on a final peg parse failure instead of throwing the raw parser position and input. keep the raw arguments string in func_args_not_string when it is not valid json instead of aborting the prompt render. * chat: surface peg-native parse failures a final peg parse failure threw the raw parser position and input. log the unparsed fragment and raise a clearer error instead, so a model output that does not match the expected format no longer fails silently with an empty assistant turn. minimal change, no behavior change on successful parses. * chat: handle openai style tool calls in peg-native * nits * common: scope OpenAI wrapper grammar trigger via autoparser flag * chat: gate type:function parsing leniency on the analysis flag Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set.

* chat: harden peg-native tool call parsing accept an optional leading type: function field in build_json_tools_flat_keys so openai style tool calls parse on templates whose serialization opens on the name field. return a clean error and log the unparsed fragment on a final peg parse failure instead of throwing the raw parser position and input. keep the raw arguments string in func_args_not_string when it is not valid json instead of aborting the prompt render. * chat: surface peg-native parse failures a final peg parse failure threw the raw parser position and input. log the unparsed fragment and raise a clearer error instead, so a model output that does not match the expected format no longer fails silently with an empty assistant turn. minimal change, no behavior change on successful parses. * chat: handle openai style tool calls in peg-native * nits * common: scope OpenAI wrapper grammar trigger via autoparser flag * chat: gate type:function parsing leniency on the analysis flag Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set. (cherry picked from commit 581e8ec)

ServeurpersoCom requested a review from a team as a code owner June 9, 2026 07:39

aldehir reviewed Jun 9, 2026

View reviewed changes

Comment thread common/chat.cpp Outdated

Comment thread common/chat.cpp Outdated

Comment thread common/chat-peg-parser.cpp Outdated

ServeurpersoCom added 2 commits June 9, 2026 20:23

chat: handle openai style tool calls in peg-native

6dd339e

nits

d56e988

aldehir reviewed Jun 9, 2026

View reviewed changes

Comment thread common/chat-auto-parser-generator.cpp Outdated

common: scope OpenAI wrapper grammar trigger via autoparser flag

c2cfb28

aldehir approved these changes Jun 9, 2026

View reviewed changes

chat: gate type:function parsing leniency on the analysis flag

07b23d6

Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set.

pwilkin approved these changes Jun 15, 2026

View reviewed changes

ServeurpersoCom merged commit 581e8ec into ggml-org:master Jun 15, 2026
24 of 25 checks passed

pwilkin mentioned this pull request Jun 15, 2026

chat: include full unparsed prompt in debug message on parse error #24650

Merged

thecodeassassin mentioned this pull request Jun 15, 2026

Eval bug: peg-native: tool-call grammar generator emits GBNF that fails its own parser (Kimi K2.7-Code, 22-tool fan-out) #24658

Open

devnen mentioned this pull request Jun 15, 2026

Eval bug: Step 3.7 Flash gets stuck in reasoning trying to make tool calls (autoparser) #24181

Closed

wollastonzhu mentioned this pull request Jun 16, 2026

server: fix checkpoints creation #22929

Merged

dhavaltechbudgie mentioned this pull request Jun 18, 2026

Eval bug: Qwen 3.6 27B forcing full prompt re-processing due to lack of cache data #22746

Open

TheTom mentioned this pull request Jul 2, 2026

fix: cherry-pick 16 upstream P0/high-P1 fixes (CUDA FA crashes + overflows, server DoS, OOB reads, tool-call parsing) TheTom/llama-cpp-turboquant#202

Merged

Uh oh!

Conversation

ServeurpersoCom commented Jun 9, 2026

Overview

Additional information

Requirements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pwilkin commented Jun 9, 2026

Uh oh!

ServeurpersoCom commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldehir commented Jun 9, 2026

Uh oh!

ServeurpersoCom commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented Jun 9, 2026

Uh oh!

aldehir commented Jun 9, 2026

Uh oh!

ServeurpersoCom commented Jun 9, 2026

Uh oh!

ServeurpersoCom commented Jun 9, 2026

Uh oh!

Uh oh!

aldehir Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

aldehir commented Jun 15, 2026

Uh oh!

pwilkin left a comment

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented Jun 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ServeurpersoCom commented Jun 9, 2026 •

edited

Loading

ServeurpersoCom commented Jun 9, 2026 •

edited

Loading