[Bug] can NOT launch ChatQnA on xeon using K8S manifest files

[lianhao]

Priority

P1-Stopper

OS type

Ubuntu

Hardware type

Xeon-SPR

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

The version with PR #669 merged.

Description

Launch ChatQnA on xeon by the following command:

kubectl install -n lianhao-test -f ChatQnA/kubernetes/manifests/xeon/chatqna.yaml

Then check the pod status by the followings:

$ kubectl -n lianhao-test get pod
NAME                                        READY   STATUS             RESTARTS        AGE
chatqna-69b57f4976-p4glm                    1/1     Running            0               7m39s
chatqna-data-prep-5cc8f86b77-plm9s          1/1     Running            0               7m40s
chatqna-embedding-usvc-5898b9766d-7fqzx     1/1     Running            0               7m40s
chatqna-llm-uservice-58ffcbf4d4-gkddl       1/1     Running            0               7m40s
chatqna-nginx-deployment-6555fc5ff7-mjlk7   0/1     CrashLoopBackOff   6 (113s ago)    7m39s
chatqna-redis-vector-db-676fb75667-kqlc2    1/1     Running            0               7m40s
chatqna-reranking-usvc-844dcf794b-sj2c7     1/1     Running            0               7m40s
chatqna-retriever-usvc-6f569d56d5-dr25s     1/1     Running            1 (7m34s ago)   7m40s
chatqna-tei-67b6b789db-n8vqw                1/1     Running            0               7m40s
chatqna-teirerank-657b8b9464-pnkw4          1/1     Running            0               7m40s
chatqna-tgi-ddc6696cb-2pmrw                 1/1     Running            0               7m40s
chatqna-ui-9b85b64d9-svwzd                  1/1     Running            0               7m40s

nginx pod keeps crashing. The root cause of this that in nginx configuration, the upstream URL is hardcoded, which doesn't support install ChatQnA in non-default k8s namespace.

Reproduce steps

kubectl create ns lianhao-test
kubectl install -n lianhao-test -f ChatQnA/kubernetes/manifests/xeon/chatqna.yaml
kubectl -n lianhao-test get pod

Raw log

No response