[ChatQnA] Switch to vLLM as default llm backend on Xeon

[wangkl2]

Description

Switch to vLLM as the default LLM backend on Xeon for ChatQnA pipeline.

Switching from TGI to vLLM as the default LLM serving backend on Xeon for the ChatQnA example to enhance the perf. Via benchmarking on Xeon server with vLLM and TGI backend for LLM component for different ISL/OSL and various number of queries and concurrency, the geomean of measured LLMServe perf on a 7B model shows perf improvement of vLLM over TGI on several metrics including average total latency, average TTFT, average TPOT and throughput. TGI is still offered as an option to deploy for LLM serving. Besides, vLLM LLM also replaces TGI LLM for other provided E2E ChatQnA pipelines including without-rerank pipeline, pinecone as the vectorDB, qdrant as the vectorDB. This PR also aligns the parameters of llm service in all chatqna test scripts with what in readme file.

Issues

#1213

Type of change

• New feature (non-breaking change which adds new functionality)
• Others (enhancement, documentation, validation, etc.)

Dependencies

n/a

Tests

TGI version: 2.4.0
vLLM version: 0.6.6.post2.dev151+gbd828722

Benchmark and compare the LLMServe perf on GNR server with OOB-vLLM and OOB-TGI backend via GenAIEval. The geomean perf of vLLM performs better than TGI for average total latency, average TTFT, average TPOT and throughput on 7B LLM with 4 sets of ISL/OSL (128/128, 128/1024, 1024/128, 1024/1024), measured on different num_queries and concurrency, including 32/8, 128/32.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files