Cannot work with qwen3.6-35b-int4

[wududu123]

run model cmd:

docker run -d
--name qwen3.6-35b-int4
--gpus all
--ipc=host
--shm-size=16g
-p 8000:8000
-v ~/.cache/modelscope/hub/models/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit:/model
-e VLLM_USE_MODELSCOPE=True
vllm/vllm-openai:latest
--model /model
--tensor-parallel-size 2
--max-model-len 24576
--gpu-memory-utilization 0.90
--host 0.0.0.0
--port 8000
--enable-auto-tool-choice
--tool-call-parser hermes

--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

When I send cmd, the server get error:

(APIServer pid=1) INFO: 172.17.0.1:52416 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] Error in extracting tool call from response.
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] Traceback (most recent call last):
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] File "/usr/local/lib/python3.12/dist-packages/vllm/tool_parsers/hermes_tool_parser.py", line 114, in extract_tool_calls
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] json.loads(match[0] if match[0] else match[1])
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] File "/usr/lib/python3.12/json/init.py", line 346, in loads
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] return _default_decoder.decode(s)
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] File "/usr/lib/python3.12/json/decoder.py", line 338, in decode
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] obj, end = self.raw_decode(s, idx=_w(s, 0).end())
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] File "/usr/lib/python3.12/json/decoder.py", line 356, in raw_decode
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] raise JSONDecodeError("Expecting value", s, err.value) from None
(APIServer pid=1) ERROR 05-26 02:20:09 [hermes_tool_parser.py:139] json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
(APIServer pid=1) INFO: 172.17.0.1:34668 - "POST /v1/chat/completions HTTP/1.1" 200 OK

With mtp or not, got the same error.

[nonlooped]

This looks like a vLLM/Qwen tool call issue than a smallcode issue, it reaches the server, but vLLM is failing trying to parse the model output as a tool call.

The issue is that the reasoning block from qwen3.6 is being passed directly to the Hermes tool parser, which needs valid JSON so you need --reasoning-parser to remove the thinking content before the tool call parser sees it.

I was reading their docs and recommended fixes and I think these are the correct flags:
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

[Doorman11991]

This is a vLLM configuration issue, not a SmallCode bug. The error occurs because vLLM's Hermes tool parser receives the raw model output including the Qwen3 reasoning block (...), which isn't valid JSON.

Fix — add these flags to your vLLM launch command:

�ash --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder

The --reasoning-parser qwen3 flag strips the thinking block before the tool call parser sees it. Without it, the Hermes parser tries to parse ...{...} as JSON and fails.

Your full corrected command:
\\�ash
docker run -d
--name qwen3.6-35b-int4
--gpus all
--ipc=host
--shm-size=16g
-p 8000:8000
-v ~/.cache/modelscope/hub/models/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit:/model
-e VLLM_USE_MODELSCOPE=True
vllm/vllm-openai:latest
--model /model
--tensor-parallel-size 2
--max-model-len 24576
--gpu-memory-utilization 0.90
--host 0.0.0.0
--port 8000
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
\\

Thanks to @nonlooped for the correct diagnosis in the comments.

[wududu123]

No output！

› 检查文件E:\source\Release\4.2\scripts\startup\bim\Architecture\Build\wall_type_edit.py 存在哪些bug

⚙ read_file ✓ 5527ms
⚙ read_file ✓ 2ms
⚙ read_file ✓ 1ms
⚙ read_file ✓ 1ms
⚙ read_file ✓ 1ms
⚡ Model has read 5 things without producing output — nudging toward output.
─── 5 tool calls this turn ───

server log:
APIServer pid=1) INFO 05-30 02:40:03 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:40:13 [loggers.py:271] Engine 000: Avg prompt throughput: 11.5 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO: 172.17.0.1:43872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 05-30 02:40:23 [loggers.py:271] Engine 000: Avg prompt throughput: 1538.7 tokens/s, Avg generation throughput: 89.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:40:33 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 113.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:40:43 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 112.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:40:53 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 111.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:41:03 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 111.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:41:13 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 111.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-30 02:41:23 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 111.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO: 172.17.0.1:36698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 172.17.0.1:53156 - "POST /v1/chat/completions HTTP/1.1" 200 OK

deploy cmd:

docker run -d
--name qwen3.6-35b-int4
--gpus all
--ipc=host
--shm-size=16g
-p 8000:8000
-v ~/.cache/modelscope/hub/models/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit:/model
-e VLLM_USE_MODELSCOPE=True
vllm/vllm-openai:latest
--model /model
--pipeline-parallel-size 2
--max-model-len 262144
--gpu-memory-utilization 0.90
--host 0.0.0.0
--port 8000
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder

.env:

✅ 指向你的私有推理服务， Hidden

SMALLCODE_BASE_URL=http://myserver:xxxx/v1

✅ 必须和你 curl 里的 model 一致

SMALLCODE_MODEL=/model

❌ 不需要 API Key（如果服务没开鉴权）

如果后面报 401，再补一行：

SMALLCODE_API_KEY=xxx

可选：限制上下文，防止爆显存

SMALLCODE_CONTEXT_BUDGET=4096

[Doorman11991]

I apologize that you're having issues, I'm working on your issue as we speak.