[Bug] default ChatQnA's model Intel/neural-chat-7b-v3-3 is extremely slow with vLLM on ICX cpu

[lianhao]

Priority

P4-Low

OS type

Ubuntu

Hardware type

Xeon-ICX

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

docker compose commit id: 742cb6d
docker image opea/vllm info:
- repoDigest: opea/vllm@sha256:61760224596acb8fbce25dfd4942049263363764f327d6d6ea8e1e69c0799988
- "Created": "2025-01-17T03:57:18.181292698Z"

Description

PR #1403 switch to vLLM as the default inference backend for ChatQnA for xeon cpu environment, however, the ChatQnA's Intel/neural-chat-7b-v3-3 model is extremely slow on vLLM on Xeon ICX CPU.

Run the following curl command to inference with vLLM directly to generate 32 tokens, it takes more than 4minutes to complete:
curl http://localhost:9009/v1/completions -H "Content-Type: application/json" -d '{"model": "Intel/neural-chat-7b-v3-3", "prompt": "What is Deep Learning?", "max_tokens": 32}'

We should either switch to another model or revert the PR #1403 to use TGI as the default inference backend on Xeon CPU.

Reproduce steps

docker compose up -d
curl http://localhost:9009/v1/completions -H "Content-Type: application/json" -d '{"model": "Intel/neural-chat-7b-v3-3", "prompt": "What is Deep Learning?", "max_tokens": 32}'

Raw log

[wangkl2]

Confirmed with Lianhao:

• The workload was deployed on ICX, instead of SPR, which does not support amx_bf16 ISA.
• vLLM version in opea/vllm:latest docker image:
  INFO 01-20 03:47:35 api_server.py:768] vLLM API server version 0.6.6.post2.dev246+gebc73f28
• The reported issue only occurred on Intel/neural-chat-7b-v3-3, while it only takes seconds to response for Qwen/Qwen2.5-Coder-7B-Instruct.

Will look into this, and update the progress later.

[wangkl2]

@lianhao I cannot reproduce this error on my end, either using the vllm version 0.6.6.post2.dev151+gbd828722 (what I used for benchmarking with the latest commit in last week in the #1403 PR) or the vllm version 0.6.6.post2.dev246+gebc73f28 (what the current opea/vllm:latest docker image is using).
For either Intel/neural-chat-7b-v3-3 or Qwen/Qwen2-7B-Instruct with vLLM backend on a GNR-SP node with BF16, it takes less than 5 seconds to respond for LLM microservice validation.

[lianhao]

vllm.log

attached is the docker logs of the same version vllm running on ICX(Xeon Platinum 8368 with 1TB memory) with model Intel/neural-chat-7b-v3-3

[wangkl2]

From the log, the compute dtype is not specified and it should use FP32, utilizing AVX512_FP32 on ICX platform.

As verified, this reported vllm perf regression issue currently only occurs on ICX with neural-chat-7b model, while it works well for other models on ICX or models including neural-chat 7b on 4th/5th/6th Xeon platform.

[chensuyue]

From the log, the compute dtype is not specified and it should use FP32, utilizing AVX512_FP32 on ICX platform.

As verified, this reported vllm perf regression issue currently only occurs on ICX with neural-chat-7b model, while it works well for other models on ICX or models including neural-chat 7b on 4th/5th/6th Xeon platform.

How about update the model? It's in the plan.

[wangkl2]

From the log, the compute dtype is not specified and it should use FP32, utilizing AVX512_FP32 on ICX platform.
As verified, this reported vllm perf regression issue currently only occurs on ICX with neural-chat-7b model, while it works well for other models on ICX or models including neural-chat 7b on 4th/5th/6th Xeon platform.

How about update the model? It's in the plan.

I‘m preparing the PR to update the default LLM.