Skip to content

Fix vllm model cache directory#1642

Merged
lvliang-intel merged 4 commits into
opea-project:mainfrom
wangkl2:fix-vllm-model-cache
Mar 10, 2025
Merged

Fix vllm model cache directory#1642
lvliang-intel merged 4 commits into
opea-project:mainfrom
wangkl2:fix-vllm-model-cache

Conversation

@wangkl2
Copy link
Copy Markdown
Collaborator

@wangkl2 wangkl2 commented Mar 10, 2025

Description

The LLM will be downloaded and saved under /root/.cache/huggingface/hub in vLLM docker container, using -v ./data:/data would cause the LLM to be downloaded again instead of loading from cache.

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

Dependencies

n/a

Tests

Local tests

wangkl2 and others added 4 commits March 7, 2025 02:39
@wangkl2
Fix the model cache dir in vllm container
76cb736 
The LLM will be downloaded and saved under `/root/.cache/huggingface/hub` in vLLM docker container, using `-v ./data:/data` would cause the LLM to be downloaded again instead of loading from cache

Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Update for CodeTrans
7f4d5de 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@wangkl2
Merge branch 'opea-project:main' into fix-vllm-model-cache
d7a6bf9
@wangkl2
Update for faqgen
3ca95c5 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 10, 2025

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

yinghu5
yinghu5 approved these changes Mar 10, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@yinghu5 yinghu5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@lvliang-intel lvliang-intel merged commit 5362321 into opea-project:main Mar 10, 2025
jedwards-habana pushed a commit to jedwards-habana/GenAIExamples that referenced this pull request Mar 11, 2025
@wangkl2
Fix vllm model cache directory (opea-project#1642)
a5f397d 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: Edwards, James A <jaedwards@habana.ai>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request Mar 21, 2025
@wangkl2
Fix vllm model cache directory (opea-project#1642)
304d835 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request Apr 1, 2025
@wangkl2
Fix vllm model cache directory (opea-project#1642)
e6b91be 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request Apr 1, 2025
@wangkl2
Fix vllm model cache directory (opea-project#1642)
b34193c 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/OPEA-GenAIExamples that referenced this pull request May 16, 2025
@wangkl2 @chyundunovDatamonsters
Fix vllm model cache directory (opea-project#1642)
a6d6f1f 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: Chingis Yundunov <c.yundunov@datamonsters.com>
letonghan pushed a commit that referenced this pull request Sep 17, 2025
@ZePan110
Remove invalid codeowner. (#1642)
c8b7e3c 
Signed-off-by: ZePan110 <ze.pan@intel.com>
letonghan pushed a commit that referenced this pull request Sep 17, 2025
@alexsin368 @pre-commit-ci
@yinghu5 @ZePan110 @lvliang-intel @MSCetin37 @RazvanLiviuVarzaru @chensuyue @Spycsh
Agent: add support for remote server (#1644)
fcf7a5a 
* add support for remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* add steps to enable remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove use_remote_service

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add OpenAI models instructions, fix format of commands

Signed-off-by: alexsin368 <alex.sin@intel.com>

* simplify ChatOpenAI instantiation

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "simplify ChatOpenAI instantiation"

This reverts commit b7c4acf7d397a284f1499254fa8832533c0c98e3.

* add back check and logic for llm_engine, set openai_key argument

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Provide ARCH option for lvm-video-llama image build (#1630)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Add sglang microservice for supporting llama4 model (#1640)

Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Co-authored-by: Lv,Liang1 <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Remove invalid codeowner. (#1642)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* add support for remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* add steps to enable remote server

Signed-off-by: alexsin368 <alex.sin@intel.com>

* remove use_remote_service

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: alexsin368 <alex.sin@intel.com>

* bug fix for chunk_size and overlap cause error in dataprep ingestion (#1643)

* bug fix for dataingest url

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* add validation function

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* validation update

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update validation function

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* MariaDB Vector integrations for retriever & dataprep services (#1645)

* Add MariaDB Vector third-party service

MariaDB Vector was introduced since MariaDB Server 11.7

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add retriever MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* Add dataprep MariaDB Vector integration

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI failures

- md5 is used for the primary key not as a security hash
- fixed mariadb readme headers

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>

---------

Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* update PR reviewers (#1651)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Expand test matrix, find all tests use 3rd party Dockerfiles (#1676)

* Expand test matrix, find all tests use 3rd party Dockerfiles

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* fix the typo of README.md Comp (#1679)

Update README.md for first entry of OPEA

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix request handle timeout issue (#1687)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* FEAT: Enable OPEA microservices to start as MCP servers (#1635)

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix huggingface_hub API upgrade issue (#1691)

* Fix huggingfacehub API upgrade issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* add OpenAI models instructions, fix format of commands

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix dataprep opensearch ingest issue (#1697)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* Fix embedding issue with ArangoDB due to deprecated HuggingFace API (#1694)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: alexsin368 <alex.sin@intel.com>

* simplify ChatOpenAI instantiation

Signed-off-by: alexsin368 <alex.sin@intel.com>

* Revert "simplify ChatOpenAI instantiation"

This reverts commit b7c4acf7d397a284f1499254fa8832533c0c98e3.

Signed-off-by: alexsin368 <alex.sin@intel.com>

* add back check and logic for llm_engine, set openai_key argument

Signed-off-by: alexsin368 <alex.sin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: alexsin368 <alex.sin@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ye, Xinyu <xinyu.ye@intel.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ying Hu <ying.hu@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: Razvan Liviu Varzaru <45736827+RazvanLiviuVarzaru@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Spycsh <39623753+Spycsh@users.noreply.github.com>
cogniware-devops pushed a commit to Cogniware-Inc/GenAIExamples that referenced this pull request Dec 19, 2025
@wangkl2 @cogniware-devops
Fix vllm model cache directory (opea-project#1642)
6dc579f 
Signed-off-by: Wang, Kai Lawrence <kai.lawrence.wang@intel.com>
Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yinghu5 yinghu5 yinghu5 approved these changes

@lvliang-intel lvliang-intel lvliang-intel approved these changes

@letonghan letonghan letonghan approved these changes

@XinyaoWa XinyaoWa Awaiting requested review from XinyaoWa

@Spycsh Spycsh Awaiting requested review from Spycsh Spycsh is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wangkl2 @yinghu5 @lvliang-intel @letonghan