Enable vllm for DocSum by letonghan · Pull Request #1716 · opea-project/GenAIExamples

letonghan · 2025-03-25T05:46:51Z

Description

Set vllm as default llm serving, and add related docker compose files, readmes, and test scripts.

Issues

#1436

Type of change

List the type of change like below. Please delete options that are not relevant.

New feature (non-breaking change which adds new functionality)

Dependencies

None

Tests

local tested

Signed-off-by: letonghan <letong.han@intel.com>

github-actions · 2025-03-25T05:47:04Z

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

for more information, see https://pre-commit.ci

Signed-off-by: letonghan <letong.han@intel.com>

into vllm_docsum

Signed-off-by: letonghan <letong.han@intel.com>

letonghan · 2025-03-26T05:40:16Z

@XinyaoWa xinyao will help to check the DocSum tgi issues of max tokens/langchain dependency versions.

…into vllm_docsum

Signed-off-by: letonghan <letong.han@intel.com>

lkk12014402

LGTM

Signed-off-by: letonghan <letong.han@intel.com>

Set vllm as default llm serving, and add related docker compose files, readmes, and test scripts. Fix issue opea-project#1436 Signed-off-by: letonghan <letong.han@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>

Set vllm as default llm serving, and add related docker compose files, readmes, and test scripts. Fix issue opea-project#1436 Signed-off-by: letonghan <letong.han@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Chingis Yundunov <c.yundunov@datamonsters.com>

* initial file structure created. Populated with unimplemented files Signed-off-by: Madison Evans <madison.evans@intel.com> * added relevant code to files within comps/router/deployment Signed-off-by: Madison Evans <madison.evans@intel.com> * added Dockerfile, opea_router_microservice.py, README.md, and requirements.txt contents Signed-off-by: Madison Evans <madison.evans@intel.com> * added controller components for router instances Signed-off-by: Madison Evans <madison.evans@intel.com> * added initial routellm controller test script in router directory Signed-off-by: Madison Evans <madison.evans@intel.com> * fixed requirements.txt issue Signed-off-by: Madison Evans <madison.evans@intel.com> * added HUGGINGFACEHUB_API_TOKEN as an env variable Signed-off-by: Madison Evans <madison.evans@intel.com> * removed hard OPENAI dependency and made OPENAI_API_KEY default to empty str Signed-off-by: Madison Evans <madison.evans@intel.com> * removed empty str fallback for OPENAI_API_KEY var Signed-off-by: Madison Evans <madison.evans@intel.com> * target localhost in RouteLLM E2E test to avoid Docker network issues Signed-off-by: Madison Evans <madison.evans@intel.com> * fixed e2e test issue for routellm test Signed-off-by: Madison Evans <madison.evans@intel.com> * changed the checkpoint path for the custom mf model weights. Now using 'routellm-e5-base-V2' under OPEA HF group Signed-off-by: Madison Evans <madison.evans@intel.com> * moved RouteEndpointDoc class into 'api_protocol.py' under cores/proto Signed-off-by: Madison Evans <madison.evans@intel.com> * added 'router-compose.yaml' to workflows/docker/compose Signed-off-by: Madison Evans <madison.evans@intel.com> * pre commit format updates Signed-off-by: Madison Evans <madison.evans@intel.com> * removed the forked version of RouteLLM from requirements.txt dependency. Now pulls from the referenced repo and then applies the patch located at 'comps/router/src/hf_compatibility.patch' Signed-off-by: Madison Evans <madison.evans@intel.com> * updated README to reflect the patch usage for modified RouteLLM repo Signed-off-by: Madison Evans <madison.evans@intel.com> * added H1 title to README Signed-off-by: Madison Evans <madison.evans@intel.com> * comply with formatting requests. Signed-off-by: Haim Barad <haim.barad@intel.com> * fix pre-commit issues: remove trailing whitespace and add newline Signed-off-by: Haim Barad <haim.barad@intel.com> --------- Signed-off-by: Madison Evans <madison.evans@intel.com> Signed-off-by: Haim Barad <haim.barad@intel.com> Co-authored-by: Haim Barad <haim.barad@intel.com>

Set vllm as default llm serving, and add related docker compose files, readmes, and test scripts. Fix issue opea-project#1436 Signed-off-by: letonghan <letong.han@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>

letonghan added 3 commits March 24, 2025 11:43

refactor files of tgi on xeon

433783c

Signed-off-by: letonghan <letong.han@intel.com>

support vllm on xeon

46d795d

Signed-off-by: letonghan <letong.han@intel.com>

support vllm in DocSum gaudi and refine related files

0c44b7e

Signed-off-by: letonghan <letong.han@intel.com>

letonghan requested a review from XinyaoWa as a code owner March 25, 2025 05:46

pre-commit-ci Bot and others added 2 commits March 25, 2025 05:47

[pre-commit.ci] auto fixes from pre-commit.com hooks

5cc148d

for more information, see https://pre-commit.ci

Merge branch 'main' into vllm_docsum

a6a9965

letonghan requested a review from lvliang-intel March 25, 2025 05:57

letonghan and others added 2 commits March 25, 2025 14:03

update expected results of long text tests

42a9df0

Signed-off-by: letonghan <letong.han@intel.com>

Merge branch 'main' into vllm_docsum

9269923

eero-t mentioned this pull request Mar 25, 2025

Added docsum service as a part of stresscli opea-project/GenAIEval#252

Merged

letonghan and others added 4 commits March 25, 2025 22:49

refine test case

f4febe7

Signed-off-by: letonghan <letong.han@intel.com>

Merge branch 'vllm_docsum' of https://github.com/letonghan/GenAIExamples

e6339e0

into vllm_docsum

fix typo

180d55e

Signed-off-by: letonghan <letong.han@intel.com>

Merge branch 'main' into vllm_docsum

f1fabdf

XinyaoWa mentioned this pull request Mar 27, 2025

Enlarge DocSum prompt buffer opea-project/GenAIComps#1471

Merged

4 tasks

letonghan added 4 commits March 27, 2025 21:50

Merge branch 'main' of https://github.com/opea-project/GenAIExamples …

0f7fa1d

…into vllm_docsum

modify to short file for type=stuff in test scripts

a12ccd5

Signed-off-by: letonghan <letong.han@intel.com>

update expected results

0471f96

Signed-off-by: letonghan <letong.han@intel.com>

fix typo

aae336f

Signed-off-by: letonghan <letong.han@intel.com>

XinyaoWa approved these changes Mar 28, 2025

View reviewed changes

lkk12014402 approved these changes Mar 28, 2025

View reviewed changes

Spycsh reviewed Mar 28, 2025

View reviewed changes

Comment thread DocSum/docker_compose/intel/hpu/gaudi/README.md Outdated

fix typo in readme

49c8074

Signed-off-by: letonghan <letong.han@intel.com>

Spycsh reviewed Mar 28, 2025

View reviewed changes

Comment thread DocSum/docker_compose/intel/hpu/gaudi/compose.yaml Outdated

refine healthcheck endpoint to localhost:80

15777e0

Signed-off-by: letonghan <letong.han@intel.com>

letonghan merged commit d4dcbd1 into opea-project:main Mar 28, 2025

letonghan mentioned this pull request Mar 31, 2025

[Feature] vLLM enablement for 8 GenAI examples #1436

Closed

21 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable vllm for DocSum#1716

Enable vllm for DocSum#1716
letonghan merged 17 commits into
opea-project:mainfrom
letonghan:vllm_docsum

letonghan commented Mar 25, 2025

Uh oh!

github-actions Bot commented Mar 25, 2025 •

edited

Loading

Uh oh!

letonghan commented Mar 26, 2025

Uh oh!

lkk12014402 left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

letonghan commented Mar 25, 2025

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

github-actions Bot commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Scanned Files

Uh oh!

letonghan commented Mar 26, 2025

Uh oh!

lkk12014402 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Mar 25, 2025 •

edited

Loading