Model too slow and I cant wirte anithing

[LeLaboDuGame]

Hello I use a q4_0 13B of llama on Hugging Face : https://huggingface.co/decapoda-research/llama-13b-hf
My config is:
16go ram
i5 10400f
rx6000xt amd
Windows 11 pro

I use llama on python because on llama.cpp I cant write any letter (my mouse disapear on the terminal and I cant wrote) and when I want to do a completion this take something like 10 min to resolve

Python code:

from llama_cpp import Llama

llm = Llama("D:\\ia\\ia\\llama.cpp\\models\\13B\\ggml-model-q4_0-llama-13B.bin")
while True:
    output = llm(input(">>> "), max_tokens=100)
    print(output)

Terminal:

llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =  73.73 KB
llama_model_load_internal: mem required  = 9807.47 MB (+ 1608.00 MB per state)
llama_init_from_file: kv self size  =  400.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
>>> Hey !

I dont know why is very slow when I load the model (1min) and why when I do a completion it takes a lot of time (~+10min) and why I can't write anything with llama.cpp.
Xd a lot of probleme !

I hope you know a solution !
Thanks in advance !
Labo.

PS:I'm not very good in english ^^

[Azeirah]

Hi!

Improving performance

Running a large language model quickly requires powerful hardware. Your hardware is pretty good but it's not amazing when it comes to LLMs.

The first thing I'm noticing is that you don't have BLAS in your binary:

AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

BLAS = 0

With recent updates to llama.cpp your GPU can be used to speed-up the model significantly. Since you have an AMD GPU, you should use the release with clblast (there are different releases for different GPUs. You need clblast, not cublas or openblas)

Also, when you're using llama from python, I think it's not using the same one as you install from the zip file in the releases, although I'm not 100% sure since I don't use Windows. So you'll be missing out on the aforementioned speed benefits.

It's still going to be slow

That said, your computer is not amazingly fit for running LLMs. You have barely enough RAM to fit in the 4-bit quantised 13B model which is not going to be great for performance. Also, do you have an SSD?

Terminal issue

As for your terminal issue, I'm not sure. You'd need to at the very least post the command you ran so we can see what's wrong. A screenshot would be helpful too.

[LeLaboDuGame]

thks I try it later !

[LeLaboDuGame]

Hi!

Improving performance

Running a large language model quickly requires powerful hardware. Your hardware is pretty good but it's not amazing when it comes to LLMs.

The first thing I'm noticing is that you don't have BLAS in your binary:

AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

BLAS = 0

With recent updates to llama.cpp your GPU can be used to speed-up the model significantly. Since you have an AMD GPU, you should use the release with clblast (there are different releases for different GPUs. You need clblast, not cublas or openblas)

Also, when you're using llama from python, I think it's not using the same one as you install from the zip file in the releases, although I'm not 100% sure since I don't use Windows. So you'll be missing out on the aforementioned speed benefits.

It's still going to be slow

That said, your computer is not amazingly fit for running LLMs. You have barely enough RAM to fit in the 4-bit quantised 13B model which is not going to be great for performance. Also, do you have an SSD?

Terminal issue

As for your terminal issue, I'm not sure. You'd need to at the very least post the command you ran so we can see what's wrong. A screenshot would be helpful too.

Ok so I'm stuck because i dont know how to dl Clblast into llama.
Can you help me ?
thanks !