fix(retriever): HybridRetriever passes unlimited documents to LLM with no token budget

[GovindhKishore]

Problem

HybridRetriever in csv_chroma.py passes all retrieved documents to create_stuff_documents_chain with zero token awareness. With two databases installed (Reactome + UniProt), this results in 60-100 documents per query reaching the LLM.

This causes three problems:

1. Verbose, unfocused answers
The LLM receives too much context including tangentially related documents. Combined with multi-query expansion across both databases, the signal-to-noise ratio degrades significantly.

2. High token cost

60-100 docs × ~150 tokens avg = 9,000-15,000 tokens per query
× 1,000 users/day = ~15M tokens/day
GPT-4o at $2.50/1M input tokens = ~$1,100/month

3. Lost in the middle
Research shows LLMs pay significantly less attention to documents in the middle of long contexts. Passing 60-100 documents means the LLM actively ignores most of them, degrading answer quality beyond a certain threshold.

Root Cause

rag_chain.py uses LangChain's create_stuff_documents_chain:

question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)

LangChain's own documentation states: "Use StuffDocumentsChain when documents are small enough to fit in context window." There is no token budgeting or document count limit applied before documents reach this chain.

Expected Behaviour

Retrieved documents should be truncated to a configurable token and count budget before being passed to the LLM. Truncation should run after ranking (WRR or FlashRank) so the least relevant documents are always removed first and the most relevant documents always survive.

Proposed Fix

Add src/util/context_truncator.py with a truncate_to_token_limit() function using tiktoken for exact token counting. Call it at the end of both retrieve_documents() and aretrieve_documents() in csv_chroma.py.

Default limits:

• max_docs: 15
• max_tokens: 12000

Both configurable via config_default.yml.

Related

• Complements PR Add FlashRank reranker to HybridRetriever to improve retrieval quality #116 (FlashRank reranking) - truncation runs after reranking so quality ordering is preserved
• Complements PR Improve response precision by refining system prompts for Reactome and UniProt retrievers #121 (prompt precision) - together these three changes form a complete solution to the verbosity problem raised by the mentor

[mithil27360]

This is a strong improvement ,adding token-aware truncation after reranking is exactly the right place in the pipeline.

One suggestion that could further improve quality:

Currently truncation is based on token count and document order (best-first). However, in practice, some documents may still contribute very little useful information even if they are ranked highly.

It might be useful to also consider:

• A minimum relevance score threshold after reranking
• Or lightweight filtering (example : remove documents with very low semantic similarity)

This could help improve signal-to-noise ratio beyond just limiting context size.

Also, since "lost in the middle" is a concern, ensuring that the most relevant documents are always placed at the beginning of the context is critical , which this implementation already aligns with.

Overall, this is a solid step toward improving both efficiency and answer quality.

[GovindhKishore]

Thanks for the suggestion @mithil27360. A minimum relevance score threshold is a good idea and worth exploring as a follow-up improvement. I kept this PR (#139) focused on the token budget and count cap to keep the scope contained. I also have PR #116 open which adds FlashRank reranking, and once both are settled score-based filtering could layer on top of that pipeline cleanly.