common/peg : implement ac parser for stricter grammar generation

[aldehir]

Overview

Even after #24839, users are still seeing the model escape the grammar. It's because the exclusion grammar can accept a partial prefix of the delimiter and each use of until(delim) is typically followed by a literal(delim).

This PR uses the same AC type and adds an including variant: consume all characters up to and including a delimiter. This way it terminates on first occurrence of \n</parameter>\n.

Fixes #24863

Additional information

New parser:

// Wraps a child parser but emits a GBNF grammar built from the Aho-Corasick
// automaton of `delimiters`, matching everything up to and including the
// first delimiter. Parsing delegates entirely to the child, which is
// responsible for consuming the delimiter (e.g. until(D) + literal(D)).
common_peg_parser ac(const common_peg_parser & p, const std::vector<std::string> & delimiters) {
    if (delimiters.empty()) {
        throw std::runtime_error("ac parser requires at least one delimiter");
    }
    return add(common_peg_ac_parser{p, delimiters});
}
common_peg_parser ac(const common_peg_parser & p, const std::string & delimiter) { return ac(p, std::vector<std::string>{delimiter}); }

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes

[jschmied]

Tested, seams to work

[mtasic85]

This might be expected behavior, but I still occasionally get in pi (0.79.9) and Qwen3.6 27B:

 <tool_call>
 <function=bash>
 <parameter=command>
 git fetch
 </parameter>
 </function>
 </tool_call>

 ./build/bin/llama-server --version
version: 9755 (0ef6f06d5)
built with GNU 16.1.1 for Linux x86_64

[aldehir]

@mtasic85 appears to be a well formed tool call. Do you see this in the content or thinking?

[mtasic85]

This is seen in the content. Thinking was disabled.

[aldehir]

@mtasic85 can you provide any verbose logs of the server when this occurs? I'm having a hard time finding a way to reproduce. I don't have a reason to believe it's related to the change here but may be related to the earlier whitespace updates.