rankAll double-applies sigmoid to Qwen3 reranker scores (already-softmaxed p_yes → range compressed to [0.5, 0.731])

[jasikpark]

Summary

For Qwen3-family rerankers (which carry a 2-class yes/no classifier head), LlamaRankingContext.rankAll() applies sigmoid() to a value that llama.cpp has already softmaxed, double-transforming the score. Relevant documents come back at ~0.73 and irrelevant at ~0.50, instead of the expected ~1.0 / ~0.0 — the usable range collapses from [0, 1] to roughly [0.5, 0.731].

Observed

Same model (ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF), same query ("what is the capital of France") + docs, through node-llama-cpp rankAll vs. llama-server's /v1/rerank — plus sigmoid() applied to the llama-server score:

doc	llama-server /v1/rerank	rankAll	sigmoid(server)
Paris is the capital of France.	0.9971	0.7306	0.7305
The Eiffel Tower is a landmark located in Paris.	0.0072	0.5010	0.5018
Photosynthesis converts sunlight into chemical energy.	0.0000	0.5000	0.5000
I had a turkey sandwich for lunch and it was fine.	0.0000	0.5000	0.5000

The rankAll column matches sigmoid(server) on every row — rankAll is returning sigmoid() of the value llama-server already returns. (The small residual, e.g. 0.5010 vs 0.5018, is run-to-run numerical noise: the two columns are two independent forward passes of the same GGUF — llama-server's build and node-llama-cpp's bundled llama.cpp — so the underlying p_yes each one sigmoids differs by ~0.001–0.003. The transform relationship is exact; the inputs to it differ by inference noise.) Since llama-server's output for this model is already a calibrated probability, the extra sigmoid is a redundant transform that compresses the usable range.

Reproduction

Minimal (node-llama-cpp only) — shows the compressed range:

import {getLlama, resolveModelFile} from "node-llama-cpp";

const llama = await getLlama();
const model = await llama.loadModel({
  modelPath: await resolveModelFile("hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf"),
});
const ctx = await model.createRankingContext({contextSize: 2048});

console.log(await ctx.rankAll("what is the capital of France", [
  "Paris is the capital of France.",                        // relevant
  "Photosynthesis converts sunlight into chemical energy.", // irrelevant
]));
// → [0.7306, 0.5000]   (expected ~[1.0, 0.0])

Full A/B script that produced the table above (node-llama-cpp vs. llama-server)

Start llama-server with the same GGUF (this is the reference "correct" scale):

llama-server -hf ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF --reranking --pooling rank --port 8090

Then run (bun or node ≥ a fetch-capable version):

import {getLlama, resolveModelFile} from "node-llama-cpp";

const query = "what is the capital of France";
const docs = [
  "Paris is the capital of France.",
  "The Eiffel Tower is a landmark located in Paris.",
  "Photosynthesis converts sunlight into chemical energy.",
  "I had a turkey sandwich for lunch and it was fine.",
];

// node-llama-cpp rankAll
const llama = await getLlama();
const model = await llama.loadModel({
  modelPath: await resolveModelFile("hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf"),
});
const ctx = await model.createRankingContext({contextSize: 2048});
const nlc = await ctx.rankAll(query, docs);

// llama-server /v1/rerank (same GGUF)
const res = await fetch("http://localhost:8090/v1/rerank", {
  method: "POST", headers: {"Content-Type": "application/json"},
  body: JSON.stringify({query, documents: docs}),
});
const server = new Array(docs.length).fill(0);
for (const r of (await res.json()).results) server[r.index] = r.relevance_score;

console.log("| doc | server | rankAll | sigmoid(server) |");
for (let i = 0; i < docs.length; i++) {
  const sigServer = 1 / (1 + Math.exp(-server[i]));
  console.log(`| ${docs[i].slice(0, 40)} | ${server[i].toFixed(4)} | ${nlc[i].toFixed(4)} | ${sigServer.toFixed(4)} |`);
}
// rankAll ≈ sigmoid(server) (residual = run-to-run numerical noise between the two engines)

Root cause

For Qwen3 rerankers, llama.cpp's RANK pooling path applies ggml_soft_max over the 2-class [yes, no] logits in the graph and stores p_yes ∈ [0, 1] in the embedding buffer — there is even an explicit comment for it in llama-graph.cpp:

// softmax for qwen3 reranker
if (arch == LLM_ARCH_QWEN3 || arch == LLM_ARCH_QWEN3VL) {
    cur = ggml_soft_max(ctx0, cur);
}

(src/llama-graph.cpp, L3039-L3041 at the time of writing — line may drift; the comment is grep-stable.)

So getEmbedding(...)[0] is already p_yes. But LlamaRankingContext then sigmoids it again (src/evaluator/LlamaRankingContext.ts#L246-L253):

const embedding = this._llamaContext._ctx.getEmbedding(input.length, 1);
if (embedding.length === 0)
    return 0;

const logit = embedding[0]!;
const probability = logitToSigmoid(logit);  // <-- p_yes is treated as a raw logit

return probability;

where logitToSigmoid (L305-L307) is 1 / (1 + e^-logit).

So p_yes — already a probability — is passed through sigmoid() a second time: sigmoid(0.997) = 0.731, sigmoid(0.000) = 0.500, matching the table (modulo the run-to-run inference noise noted above).

Scope / suggested fix

The extra sigmoid is appropriate for rerankers that emit a raw logit (no in-graph normalization), but incorrect for models where llama.cpp's RANK pooling already produces a probability (Qwen3 applies softmax in-graph, gated on LLM_ARCH_QWEN3/QWEN3VL). The fix should be conditional — e.g. skip logitToSigmoid when the pooled output is already normalized, and/or expose a raw-score option so callers can match llama-server's /v1/rerank scale.

Happy to send a PR if you can point me at the preferred direction (arch/normalization check vs. a rawScore option).

---

Environment: node-llama-cpp v3.18.1, macOS (Metal), bun. Model ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF.

[jasikpark]

One note on why this likely slipped through, in case it's useful for the fix:

The only ranking test in the repo is test/modelDependent/bgeReranker/rank.test.ts, and it covers BGE — a reranker that emits a raw logit with no in-graph softmax. For that class of model logitToSigmoid() is exactly the right transform, so the test passes and looks correct. There's no test (and no benchmark) exercising a Qwen3 reranker, which is the arch where llama.cpp already softmaxes in-graph — so the double-transform never gets hit by CI.

Worth adding: even a ranking benchmark wouldn't have caught this. The extra sigmoid is monotonic, so it preserves document order and only compresses the scale. Any ranking-quality metric (NDCG / MRR / recall@k / ordering assertions) is invariant under it — the ranking is identical, just squeezed into [0.5, 0.731]. Only an assertion on the absolute score value surfaces it, and the one value-asserting test happens to be the case where the sigmoid is correct.

So a good regression guard for the fix would be a Qwen3 ranking test that asserts the scale, not just the order — e.g. a clearly-relevant doc should score well above sigmoid(1.0) ≈ 0.731 (which is the hard ceiling a Qwen3 reranker can currently reach through rankAll).

[github-actions]

🎉 This issue has been resolved in version 3.19.0 🎉

The release is available on:

• v3.19.0
• npm package (@latest dist-tag)
• GitHub release
• v3.19.0

Your semantic-release bot 📦🚀