Skip to content

[AudioQnA] Enable vLLM and set it as default LLM serving#1657

Merged
yinghu5 merged 19 commits into
opea-project:mainfrom
wangkl2:audioqna-vllm
Mar 14, 2025
Merged

[AudioQnA] Enable vLLM and set it as default LLM serving#1657
yinghu5 merged 19 commits into
opea-project:mainfrom
wangkl2:audioqna-vllm

Conversation

@wangkl2
Copy link
Copy Markdown
Collaborator

@wangkl2 wangkl2 commented Mar 13, 2025 •
edited
Loading

Description

  • Enable vLLM for AudioQnA and set it as the default LLM serving framework on both Xeon and Gaudi
  • Update the default LLM to avoid the chat template issue with neural-chat
  • Make asr/llm/tts service ports and vllm-gaudi parameters configurable in compose files
  • Modify readmes
  • Add test scripts

Issues

#1436

Type of change

List the type of change like below. Please delete options that are not relevant.

  • New feature (non-breaking change which adds new functionality)

Dependencies

n/a

Tests

Local tests

wangkl2 added 15 commits March 12, 2025 03:09
@wangkl2
[AudioQnA] Enable vLLM and set it as default LLM serving
9b41cff 
- Enable vLLM and set it as the default LLM serving framework
- Make service ports configurable in compose files
- Modify readmes
- Add test scripts

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
use vllm for multilang
f1ea0c8 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Make WHISPER_SERVER_PORT,SPEECHT5_SERVER_PORT,GPT_SOVITS_SERVER_PORT,…
9adbf9d 
…LLM_SERVER_PORT configurable in compose files

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Update the readme for xeon
69ca597 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Update the readme for xeon
ea02da2 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
add vllm to build.yaml
1502b7c 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Make WHISPER_SERVER_PORT,WHISPER_SERVER_PORT,LLM_SERVER_PORT configur…
f654319 
…able in compose file

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Set vllm as default for gaudi
95c490f 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Make vLLM parameters configurable for Gaudi
3103b56 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Update readme for gaudi
f3a1112 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
add vllm test script for xeon
2478b3f 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
add vllm test script for gaudi
631e411 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
modify the method of health check
9561b99 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
update readme
47792fc 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Update default llm to avoid chat template issue for neural-chat
ad1c869 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 13, 2025 •
edited
Loading

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

pre-commit-ci Bot and others added 3 commits March 13, 2025 07:22
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
bc7d9e9 
for more information, see https://pre-commit.ci
@wangkl2
update test scripts
58afbd5 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
update test scripts
cc66706 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@yinghu5 yinghu5 added aitce A2 ready to fix labels Mar 13, 2025
@yinghu5 yinghu5 added this to OPEA Mar 13, 2025
@yinghu5 yinghu5 added this to the v1.3 milestone Mar 13, 2025
@yinghu5 yinghu5 added the feature New feature or request label Mar 13, 2025
@wangkl2
update test scripts
9ea952b 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@yinghu5 yinghu5 self-requested a review March 14, 2025 00:50
yinghu5
yinghu5 approved these changes Mar 14, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@yinghu5 yinghu5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you a lot!

@yinghu5
Copy link
Copy Markdown
Collaborator

yinghu5 commented Mar 14, 2025

@Spycsh @WenjiaoYue Could you please help to review the PR? thank you!

@yinghu5 yinghu5 added A1 high prority and removed A2 ready to fix labels Mar 14, 2025
@yinghu5 yinghu5 merged commit 8fe1929 into opea-project:main Mar 14, 2025
@github-project-automation github-project-automation Bot moved this to Done in OPEA Mar 14, 2025
@joshuayao joshuayao mentioned this pull request Mar 14, 2025
Closed
21 tasks
@joshuayao joshuayao removed this from OPEA Mar 18, 2025
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request Mar 21, 2025
@wangkl2 @pre-commit-ci
[AudioQnA] Enable vLLM and set it as default LLM serving (opea-projec…
3c95214 
…t#1657)

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request Apr 1, 2025
@wangkl2 @pre-commit-ci
[AudioQnA] Enable vLLM and set it as default LLM serving (opea-projec…
5c21967 
…t#1657)

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request Apr 1, 2025
@wangkl2 @pre-commit-ci
[AudioQnA] Enable vLLM and set it as default LLM serving (opea-projec…
51cbac6 
…t#1657)

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request May 16, 2025
@wangkl2 @pre-commit-ci @chyundunovDatamonsters
[AudioQnA] Enable vLLM and set it as default LLM serving (opea-projec…
d12765b 
…t#1657)

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Chingis Yundunov <c.yundunov@datamonsters.com>
cogniware-devops pushed a commit to Cogniware-Inc/GenAIExamples that referenced this pull request Dec 19, 2025
@wangkl2 @pre-commit-ci @cogniware-devops
[AudioQnA] Enable vLLM and set it as default LLM serving (opea-projec…
45702f9 
…t#1657)

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yinghu5 yinghu5 yinghu5 approved these changes

@lvliang-intel lvliang-intel lvliang-intel approved these changes

@Spycsh Spycsh Awaiting requested review from Spycsh Spycsh is a code owner

@WenjiaoYue WenjiaoYue Awaiting requested review from WenjiaoYue WenjiaoYue is a code owner

Assignees

No one assigned

Labels

A1 high prority aitce feature New feature or request

Projects

None yet

Milestone

v1.3

Development

Successfully merging this pull request may close these issues.

4 participants

@wangkl2 @yinghu5 @lvliang-intel @joshuayao