move BLAS to a separate backend by slaren · Pull Request #6210 · ggml-org/llama.cpp

slaren · 2024-03-21T19:07:33Z

Moves BLAS support from ggml.c to a separate backend, and adds the necessary changes to ggml-backend to support backends that only implement matrix multiplication.

Changes to ggml-backend
- Support for fallback to CPU with ggml_backend_sched
  - Operations not implemented by a backend will be automatically run on the CPU, as long as the operation is reported as not supported in the supports_op function of the backend
- Moved buffer type function supports_backend to backend function supports_buft
- Backends that want to declare compatibility with any kind of host buffer can return ggml_backend_buft_is_host from supports_buft
- ggml_backend_sched will avoid copies between backends when the backend supports the buffer type
  - Eg. when switching from Metal to CPU, no tensors will be copied since the Metal buffers are compatible with the CPU backend (but not the other way around)
The GGML_SCHED_DEBUG environment variable can be used to view the graph splits. This is useful to see what operations are being run on each backend
Adds the BLAS backend
- Supports matrix multiplication using a BLAS library. Previously, this was supported as part of the CPU backend
- Threads are no longer spinning while BLAS is running, potentially improving performance, and batch processing is no longer limited to 4 threads when using BLAS
- The number of threads of the BLAS library configured automatically for OpenBLAS and BLIS (with -t or -tb)
- For better performance, it is recommended to use OpenMP versions of the BLAS libraries, if available (except macOS)
- Like before, to enable the BLAS backend, build with the flag LLAMA_BLAS when using cmake, or when using make, LLAMA_OPENBLAS, LLAMA_OPENBLAS64 or LLAMA_BLIS
- On macOS, this is enabled by default through Accelerate
- BLAS support has been removed from the CPU backend in ggml.c. Applications that want to support BLAS will need to use the BLAS backend
- Since this backend only implements matrix multiplication, it should be used with ggml_backend_sched alongside the CPU or other backends to provide support for other operations
- Note: the BLAS backend should not be used alongside GPU backends, as it will prevent offloading of large batches with partial offloading

ggerganov · 2024-03-21T19:38:35Z

this will also have the effect that using BLAS will require using ggml-backend and ggml_backend_sched, is that a problem?

Will just need to adapt whisper.cpp when it's ready

github-actions · 2024-05-11T08:02:46Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 556 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8392.96ms p(95)=20049.24ms fails=, finish reason: stop=505 truncated=51
Prompt processing (pp): avg=95.91tk/s p(95)=451.37tk/s
Token generation (tg): avg=32.54tk/s p(95)=46.27tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=sl/blas-backend commit=ecb75b5f54cab6ca7f77ec51eb5f7d87c87be6cd

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 556 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717788785 --> 1717789413
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 472.97, 472.97, 472.97, 472.97, 472.97, 651.19, 651.19, 651.19, 651.19, 651.19, 684.37, 684.37, 684.37, 684.37, 684.37, 717.14, 717.14, 717.14, 717.14, 717.14, 777.97, 777.97, 777.97, 777.97, 777.97, 803.49, 803.49, 803.49, 803.49, 803.49, 802.74, 802.74, 802.74, 802.74, 802.74, 830.76, 830.76, 830.76, 830.76, 830.76, 842.55, 842.55, 842.55, 842.55, 842.55, 858.83, 858.83, 858.83, 858.83, 858.83, 884.07, 884.07, 884.07, 884.07, 884.07, 906.98, 906.98, 906.98, 906.98, 906.98, 917.57, 917.57, 917.57, 917.57, 917.57, 927.93, 927.93, 927.93, 927.93, 927.93, 889.29, 889.29, 889.29, 889.29, 889.29, 888.42, 888.42, 888.42, 888.42, 888.42, 883.9, 883.9, 883.9, 883.9, 883.9, 890.66, 890.66, 890.66, 890.66, 890.66, 889.58, 889.58, 889.58, 889.58, 889.58, 885.33, 885.33, 885.33, 885.33, 885.33, 891.37, 891.37, 891.37, 891.37, 891.37, 892.18, 892.18, 892.18, 892.18, 892.18, 903.58, 903.58, 903.58, 903.58, 903.58, 884.3, 884.3, 884.3, 884.3, 884.3, 885.93, 885.93, 885.93, 885.93, 885.93, 887.7, 887.7, 887.7, 887.7, 887.7, 897.88, 897.88, 897.88, 897.88, 897.88, 898.83, 898.83, 898.83, 898.83, 898.83, 901.18, 901.18, 901.18, 901.18, 901.18, 903.85, 903.85, 903.85, 903.85, 903.85, 902.06, 902.06, 902.06, 902.06, 902.06, 899.67, 899.67, 899.67, 899.67, 899.67, 901.12, 901.12, 901.12, 901.12, 901.12, 901.61, 901.61, 901.61, 901.61, 901.61, 894.81, 894.81, 894.81, 894.81, 894.81, 899.59, 899.59, 899.59, 899.59, 899.59, 902.05, 902.05, 902.05, 902.05, 902.05, 899.29, 899.29, 899.29, 899.29, 899.29, 900.17, 900.17, 900.17, 900.17, 900.17, 901.99, 901.99, 901.99, 901.99, 901.99, 902.25, 902.25, 902.25, 902.25, 902.25, 897.96, 897.96, 897.96, 897.96, 897.96, 899.74, 899.74, 899.74, 899.74, 899.74, 898.21, 898.21, 898.21, 898.21, 898.21, 896.68, 896.68, 896.68, 896.68, 896.68, 893.88, 893.88, 893.88, 893.88, 893.88, 900.65, 900.65, 900.65, 900.65, 900.65, 900.28, 900.28, 900.28, 900.28, 900.28, 905.32, 905.32, 905.32, 905.32, 905.32, 905.04, 905.04, 905.04, 905.04, 905.04, 907.84, 907.84, 907.84, 907.84, 907.84, 911.77, 911.77, 911.77, 911.77, 911.77, 910.14, 910.14, 910.14, 910.14, 910.14, 916.43, 916.43, 916.43, 916.43, 916.43, 913.54, 913.54, 913.54, 913.54, 913.54, 913.98, 913.98, 913.98, 913.98, 913.98, 913.74, 913.74, 913.74, 913.74, 913.74, 914.78, 914.78, 914.78, 914.78, 914.78, 915.78, 915.78, 915.78, 915.78, 915.78, 917.28, 917.28, 917.28, 917.28, 917.28, 917.38, 917.38, 917.38, 917.38, 917.38]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 556 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717788785 --> 1717789413
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 40.3, 40.3, 40.3, 40.3, 40.3, 29.58, 29.58, 29.58, 29.58, 29.58, 31.08, 31.08, 31.08, 31.08, 31.08, 32.49, 32.49, 32.49, 32.49, 32.49, 32.53, 32.53, 32.53, 32.53, 32.53, 33.31, 33.31, 33.31, 33.31, 33.31, 34.81, 34.81, 34.81, 34.81, 34.81, 35.15, 35.15, 35.15, 35.15, 35.15, 35.47, 35.47, 35.47, 35.47, 35.47, 35.45, 35.45, 35.45, 35.45, 35.45, 35.54, 35.54, 35.54, 35.54, 35.54, 34.66, 34.66, 34.66, 34.66, 34.66, 32.62, 32.62, 32.62, 32.62, 32.62, 32.1, 32.1, 32.1, 32.1, 32.1, 30.91, 30.91, 30.91, 30.91, 30.91, 31.16, 31.16, 31.16, 31.16, 31.16, 31.23, 31.23, 31.23, 31.23, 31.23, 30.84, 30.84, 30.84, 30.84, 30.84, 30.62, 30.62, 30.62, 30.62, 30.62, 30.52, 30.52, 30.52, 30.52, 30.52, 30.71, 30.71, 30.71, 30.71, 30.71, 30.98, 30.98, 30.98, 30.98, 30.98, 30.9, 30.9, 30.9, 30.9, 30.9, 30.97, 30.97, 30.97, 30.97, 30.97, 31.24, 31.24, 31.24, 31.24, 31.24, 31.26, 31.26, 31.26, 31.26, 31.26, 31.41, 31.41, 31.41, 31.41, 31.41, 31.71, 31.71, 31.71, 31.71, 31.71, 31.78, 31.78, 31.78, 31.78, 31.78, 31.88, 31.88, 31.88, 31.88, 31.88, 31.92, 31.92, 31.92, 31.92, 31.92, 31.91, 31.91, 31.91, 31.91, 31.91, 31.95, 31.95, 31.95, 31.95, 31.95, 31.7, 31.7, 31.7, 31.7, 31.7, 31.31, 31.31, 31.31, 31.31, 31.31, 31.11, 31.11, 31.11, 31.11, 31.11, 31.04, 31.04, 31.04, 31.04, 31.04, 31.13, 31.13, 31.13, 31.13, 31.13, 31.3, 31.3, 31.3, 31.3, 31.3, 31.36, 31.36, 31.36, 31.36, 31.36, 31.44, 31.44, 31.44, 31.44, 31.44, 31.28, 31.28, 31.28, 31.28, 31.28, 30.79, 30.79, 30.79, 30.79, 30.79, 30.73, 30.73, 30.73, 30.73, 30.73, 29.48, 29.48, 29.48, 29.48, 29.48, 29.5, 29.5, 29.5, 29.5, 29.5, 29.6, 29.6, 29.6, 29.6, 29.6, 29.72, 29.72, 29.72, 29.72, 29.72, 29.78, 29.78, 29.78, 29.78, 29.78, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.67, 29.67, 29.67, 29.67, 29.67, 29.7, 29.7, 29.7, 29.7, 29.7, 29.6, 29.6, 29.6, 29.6, 29.6, 29.61, 29.61, 29.61, 29.61, 29.61, 29.73, 29.73, 29.73, 29.73, 29.73, 29.84, 29.84, 29.84, 29.84, 29.84, 29.92, 29.92, 29.92, 29.92, 29.92, 30.04, 30.04, 30.04, 30.04, 30.04, 30.08, 30.08, 30.08, 30.08, 30.08]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 556 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717788785 --> 1717789413
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.38, 0.38, 0.38, 0.38, 0.38, 0.2, 0.2, 0.2, 0.2, 0.2, 0.13, 0.13, 0.13, 0.13, 0.13, 0.19, 0.19, 0.19, 0.19, 0.19, 0.24, 0.24, 0.24, 0.24, 0.24, 0.09, 0.09, 0.09, 0.09, 0.09, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.17, 0.17, 0.17, 0.17, 0.17, 0.11, 0.11, 0.11, 0.11, 0.11, 0.31, 0.31, 0.31, 0.31, 0.31, 0.33, 0.33, 0.33, 0.33, 0.33, 0.44, 0.44, 0.44, 0.44, 0.44, 0.25, 0.25, 0.25, 0.25, 0.25, 0.16, 0.16, 0.16, 0.16, 0.16, 0.06, 0.06, 0.06, 0.06, 0.06, 0.31, 0.31, 0.31, 0.31, 0.31, 0.3, 0.3, 0.3, 0.3, 0.3, 0.19, 0.19, 0.19, 0.19, 0.19, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.27, 0.27, 0.27, 0.27, 0.27, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.19, 0.19, 0.19, 0.19, 0.19, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.26, 0.26, 0.26, 0.26, 0.26, 0.18, 0.18, 0.18, 0.18, 0.18, 0.17, 0.17, 0.17, 0.17, 0.17, 0.24, 0.24, 0.24, 0.24, 0.24, 0.2, 0.2, 0.2, 0.2, 0.2, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.37, 0.37, 0.37, 0.37, 0.37, 0.51, 0.51, 0.51, 0.51, 0.51, 0.48, 0.48, 0.48, 0.48, 0.48, 0.49, 0.49, 0.49, 0.49, 0.49, 0.09, 0.09, 0.09, 0.09, 0.09, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.09, 0.09, 0.09, 0.09, 0.09, 0.22, 0.22, 0.22, 0.22, 0.22, 0.26, 0.26, 0.26, 0.26, 0.26, 0.09, 0.09, 0.09, 0.09, 0.09, 0.29, 0.29, 0.29, 0.29, 0.29, 0.23, 0.23, 0.23, 0.23, 0.23, 0.17, 0.17, 0.17, 0.17, 0.17, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.2, 0.2, 0.2, 0.2, 0.2]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 556 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717788785 --> 1717789413
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 1.0, 1.0, 1.0, 1.0, 1.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0]

slaren · 2024-05-30T16:04:01Z

@mofosyne I appreciate that you are trying to help, but please don't do that on my PRs. I very often have not pushed local changes and I prefer to deal with the merge conflicts myself.

…times

slaren · 2024-06-06T00:28:32Z

@ggerganov I am thinking about how accelerate should interact with the BLAS backend. I think this would make sense:

GGML_USE_ACCELERATE defined: accelerate is used in ggml.c
GGML_USE_ACCELERATE and GGML_USE_BLAS defined: the BLAS backend is used using accelerate as the BLAS library

Conversely:

If GGML_USE_BLAS is not defined, accelerate will not be used for GEMMs

Currently llama.cpp has to check check for defined(GGML_USE_BLAS) || defined(GGML_USE_ACCELERATE) to decide when to use the BLAS backend, which doesn't seem very good. In the BLAS backend, accelerate is treated in the same way as any other BLAS library.

…ot available

ggerganov · 2024-06-06T07:04:01Z

Yes, that makes sense. With only GGML_USE_ACCELERATE we will still be able to use some non-GEMM functionality from the Accelerate framework such as vDSP - it's just very convenient for Apple Silicon devices to have this framework available in the core ggml.c. For GEMMs we would explicitly need to have GGML_USE_BLAS for all kinds of BLAS implementations, including Accelerate's BLAS

slaren · 2024-06-11T23:37:04Z

This should be good now. I have updated the PR description with more details about the changes included here.

ggerganov

Note: the BLAS backend should not be used alongside GPU backends, as it will prevent offloading of large batches with partial offloading

On macOS with Metal enabled, when I build with LLAMA_BLAS=OFF and run with partial offloading (-ngl 28), the non-offloaded layers are running on the CPU backend:

...
node # 32 (       ADD):              l_out-0 (   8M) [  CPU         ]:            ffn_out-0 (   8M) [  CPU         ]            ffn_inp-0 (   8M) [  CPU         ]
node # 33 (  RMS_NORM):               norm-1 (   8M) [  CPU         ]:              l_out-0 (   8M) [  CPU         ]
node # 34 (       MUL):          attn_norm-1 (   8M) [  CPU         ]:               norm-1 (   8M) [  CPU         ] blk.1.attn_norm.weig (  16K) [  CPU         ]
node # 35 (   MUL_MAT):               Qcur-1 (   8M) [  CPU         ]:  blk.1.attn_q.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [  CPU         ]
node # 37 (      ROPE):               Qcur-1 (   8M) [  CPU         ]:    Qcur-1 (reshaped) (   8M) [  CPU         ]              inp_pos (   2K) [  CPU         ]
node # 38 (   MUL_MAT):               Kcur-1 (   8M) [  CPU         ]:  blk.1.attn_k.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [  CPU         ]
node # 40 (      ROPE):               Kcur-1 (   8M) [  CPU         ]:    Kcur-1 (reshaped) (   8M) [  CPU         ]              inp_pos (   2K) [  CPU         ]
node # 41 (   MUL_MAT):               Vcur-1 (   8M) [  CPU         ]:  blk.1.attn_v.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [  CPU         ]
node # 43 (       CPY): k_cache_view-1 (copy (   4M) [  CPU         ]:               Kcur-1 (   8M) [  CPU         ]       k_cache_view-1 (   4M) [  CPU         ]
node # 46 (       CPY): v_cache_view-1 (copy (   4M) [  CPU         ]:  Vcur-1 (transposed) (   8M) [  CPU         ]       v_cache_view-1 (   4M) [  CPU         ]
node # 50 (   MUL_MAT):                 kq-1 (  32M) [  CPU         ]:                  k-1 (   4M) [  CPU         ]                  q-1 (   8M) [  CPU         ]
node # 51 (  SOFT_MAX):    kq_soft_max_ext-1 (  32M) [  CPU         ]:                 kq-1 (  32M) [  CPU         ]              KQ_mask (   1M) [  CPU         ]
node # 52 (   MUL_MAT):                kqv-1 (   8M) [  CPU         ]:                  v-1 (   4M) [  CPU         ]    kq_soft_max_ext-1 (  32M) [  CPU         ]
node # 54 (      CONT):    kqv_merged_cont-1 (   8M) [  CPU         ]:         kqv_merged-1 (   8M) [  CPU         ]
node # 55 (   MUL_MAT):            kqv_out-1 (   8M) [  CPU         ]: blk.1.attn_output.we (  17M) [  CPU         ]    kqv_merged_cont-1 (   8M) [  CPU         ]
node # 56 (       ADD):            ffn_inp-1 (   8M) [  CPU         ]:            kqv_out-1 (   8M) [  CPU         ]              l_out-0 (   8M) [  CPU         ]
node # 57 (  RMS_NORM):               norm-1 (   8M) [  CPU         ]:            ffn_inp-1 (   8M) [  CPU         ]
node # 58 (       MUL):           ffn_norm-1 (   8M) [  CPU         ]:               norm-1 (   8M) [  CPU         ] blk.1.ffn_norm.weigh (  16K) [  CPU         ]
node # 59 (   MUL_MAT):           ffn_gate-1 (  21M) [  CPU         ]: blk.1.ffn_gate.weigh (  45M) [  CPU         ]           ffn_norm-1 (   8M) [  CPU         ]
node # 60 (     UNARY):           ffn_silu-1 (  21M) [  CPU         ]:           ffn_gate-1 (  21M) [  CPU         ]
node # 61 (   MUL_MAT):             ffn_up-1 (  21M) [  CPU         ]:  blk.1.ffn_up.weight (  45M) [  CPU         ]           ffn_norm-1 (   8M) [  CPU         ]
node # 62 (       MUL):       ffn_gate_par-1 (  21M) [  CPU         ]:           ffn_silu-1 (  21M) [  CPU         ]             ffn_up-1 (  21M) [  CPU         ]
node # 63 (   MUL_MAT):            ffn_out-1 (   8M) [  CPU         ]: blk.1.ffn_down.weigh (  45M) [  CPU         ]       ffn_gate_par-1 (  21M) [  CPU         ]
node # 64 (       ADD):              l_out-1 (   8M) [  CPU         ]:            ffn_out-1 (   8M) [  CPU         ]            ffn_inp-1 (   8M) [  CPU         ]
node # 65 (  RMS_NORM):               norm-2 (   8M) [  CPU         ]:              l_out-1 (   8M) [  CPU         ]
...

With LLAMA_BLAS=ON it uses the BLAS backend for the matrix multiplications:

...
## SPLIT #16: Metal # 1 inputs: [ffn_out-0 (   8M)] 
node # 32 (       ADD):              l_out-0 (   8M) [Metal         ]:    Metal#ffn_out-0#0 (   8M) [ NULL         ]            ffn_inp-0 (   8M) [Metal         ]
node # 33 (  RMS_NORM):               norm-1 (   8M) [Metal         ]:              l_out-0 (   8M) [Metal         ]

## SPLIT #17: CPU # 0 inputs: 
node # 34 (       MUL):          attn_norm-1 (   8M) [  CPU         ]:               norm-1 (   8M) [Metal         ] blk.1.attn_norm.weig (  16K) [  CPU         ]

## SPLIT #18: BLAS # 0 inputs: 
node # 35 (   MUL_MAT):               Qcur-1 (   8M) [ BLAS         ]:  blk.1.attn_q.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [  CPU         ]

## SPLIT #19: Metal # 1 inputs: [Qcur-1 (reshaped) (   8M)] 
node # 37 (      ROPE):               Qcur-1 (   8M) [Metal         ]: Metal#Qcur-1 (reshap (   8M) [ NULL         ]      Metal#inp_pos#0 (   2K) [ NULL         ]

## SPLIT #20: BLAS # 0 inputs: 
node # 38 (   MUL_MAT):               Kcur-1 (   8M) [ BLAS         ]:  blk.1.attn_k.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [  CPU         ]

## SPLIT #21: Metal # 1 inputs: [Kcur-1 (reshaped) (   8M)] 
node # 40 (      ROPE):               Kcur-1 (   8M) [Metal         ]: Metal#Kcur-1 (reshap (   8M) [ NULL         ]      Metal#inp_pos#0 (   2K) [ NULL         ]

## SPLIT #22: BLAS # 0 inputs: 
node # 41 (   MUL_MAT):               Vcur-1 (   8M) [ BLAS         ]:  blk.1.attn_v.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [  CPU         ]

## SPLIT #23: CPU # 0 inputs: 
node # 43 (       CPY): k_cache_view-1 (copy (   4M) [  CPU         ]:               Kcur-1 (   8M) [Metal         ]       k_cache_view-1 (   4M) [  CPU         ]
node # 46 (       CPY): v_cache_view-1 (copy (   4M) [  CPU         ]:  Vcur-1 (transposed) (   8M) [ BLAS         ]       v_cache_view-1 (   4M) [  CPU         ]

## SPLIT #24: Metal # 2 inputs: [k-1 (   4M)] [v-1 (   4M)] 
node # 50 (   MUL_MAT):                 kq-1 (  32M) [Metal         ]:          Metal#k-1#0 (   4M) [ NULL         ]                  q-1 (   8M) [Metal         ]
node # 51 (  SOFT_MAX):    kq_soft_max_ext-1 (  32M) [Metal         ]:                 kq-1 (  32M) [Metal         ]      Metal#KQ_mask#0 (   1M) [ NULL         ]
node # 52 (   MUL_MAT):                kqv-1 (   8M) [Metal         ]:          Metal#v-1#0 (   4M) [ NULL         ]    kq_soft_max_ext-1 (  32M) [Metal         ]
node # 54 (      CONT):    kqv_merged_cont-1 (   8M) [Metal         ]:         kqv_merged-1 (   8M) [Metal         ]

## SPLIT #25: BLAS # 0 inputs: 
node # 55 (   MUL_MAT):            kqv_out-1 (   8M) [ BLAS         ]: blk.1.attn_output.we (  17M) [  CPU         ]    kqv_merged_cont-1 (   8M) [Metal         ]

## SPLIT #26: Metal # 1 inputs: [kqv_out-1 (   8M)] 
node # 56 (       ADD):            ffn_inp-1 (   8M) [Metal         ]:    Metal#kqv_out-1#0 (   8M) [ NULL         ]              l_out-0 (   8M) [Metal         ]
node # 57 (  RMS_NORM):               norm-1 (   8M) [Metal         ]:            ffn_inp-1 (   8M) [Metal         ]

## SPLIT #27: CPU # 0 inputs: 
node # 58 (       MUL):           ffn_norm-1 (   8M) [  CPU         ]:               norm-1 (   8M) [Metal         ] blk.1.ffn_norm.weigh (  16K) [  CPU         ]

## SPLIT #28: BLAS # 0 inputs: 
node # 59 (   MUL_MAT):           ffn_gate-1 (  21M) [ BLAS         ]: blk.1.ffn_gate.weigh (  45M) [  CPU         ]           ffn_norm-1 (   8M) [  CPU         ]

## SPLIT #29: Metal # 1 inputs: [ffn_gate-1 (  21M)] 
node # 60 (     UNARY):           ffn_silu-1 (  21M) [Metal         ]:   Metal#ffn_gate-1#0 (  21M) [ NULL         ]

## SPLIT #30: BLAS # 0 inputs: 
node # 61 (   MUL_MAT):             ffn_up-1 (  21M) [ BLAS         ]:  blk.1.ffn_up.weight (  45M) [  CPU         ]           ffn_norm-1 (   8M) [  CPU         ]

## SPLIT #31: Metal # 1 inputs: [ffn_up-1 (  21M)] 
node # 62 (       MUL):       ffn_gate_par-1 (  21M) [Metal         ]:           ffn_silu-1 (  21M) [Metal         ]     Metal#ffn_up-1#0 (  21M) [ NULL         ]

## SPLIT #32: BLAS # 0 inputs: 
node # 63 (   MUL_MAT):            ffn_out-1 (   8M) [ BLAS         ]: blk.1.ffn_down.weigh (  45M) [  CPU         ]       ffn_gate_par-1 (  21M) [Metal         ]

## SPLIT #33: Metal # 1 inputs: [ffn_out-1 (   8M)] 
node # 64 (       ADD):              l_out-1 (   8M) [Metal         ]:    Metal#ffn_out-1#0 (   8M) [ NULL         ]            ffn_inp-1 (   8M) [Metal         ]
node # 65 (  RMS_NORM):               norm-2 (   8M) [Metal         ]:              l_out-1 (   8M) [Metal         ]
...

Is this the expectation? It seems like using BLAS together with GPU offloading leads to improvement in this case, or did I misunderstood this comment?

slaren · 2024-06-12T08:28:06Z

Specifically, this applies to backends that implement the offload_op function to offload large batches even when the model is not completely offloaded, which currently it is CUDA, Vulkan and SYCL. For these backends, enabling the BLAS backend will cause it to be used instead of offloading large batches to the GPU by copying the weights to VRAM as needed. Since the Metal backend does not implement this function it is not affected, and the BLAS backend can be used to enable Accelerate for the layers not offloaded.

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

slaren · 2024-06-12T08:39:24Z

Metal should not be used for the operations in between the BLAS backend in not offloaded layers though, I will try to fix that.

slaren · 2024-06-12T08:55:58Z

@zhouwg I already considered it and rejected it. Spamming more about it is not going to help your cause.

ggerganov · 2024-06-12T09:38:38Z

@zhouwg Please focus on your PR and respect the comments and suggestions that have already been provided. Consider this final warning, before having to block you

zhouwg · 2024-06-12T09:42:40Z

@zhouwg Please focus on your PR and respect the comments and suggestions that have already been provided. Consider this final warning, before having to block you

thanks for your reminder. I see.

ggerganov · 2024-06-12T10:07:19Z

In that same example, if I allow the GGML_OP_MUL operation to be offloaded in the Metal backend:

diff --git a/ggml-metal.m b/ggml-metal.m
index 7786acd6..665eae15 100644
--- a/ggml-metal.m
+++ b/ggml-metal.m
@@ -3178,6 +3178,12 @@ GGML_CALL static bool ggml_backend_metal_supports_buft(ggml_backend_t backend, g
     UNUSED(backend);
 }
 
+GGML_CALL static bool ggml_backend_metal_offload_op(ggml_backend_t backend, const struct ggml_tensor * op) {
+    return (op->op == GGML_OP_MUL);
+
+    GGML_UNUSED(backend);
+}
+
 static struct ggml_backend_i ggml_backend_metal_i = {
     /* .get_name                = */ ggml_backend_metal_name,
     /* .free                    = */ ggml_backend_metal_free,
@@ -3193,7 +3199,7 @@ static struct ggml_backend_i ggml_backend_metal_i = {
     /* .graph_compute           = */ ggml_backend_metal_graph_compute,
     /* .supports_op             = */ ggml_backend_metal_supports_op,
     /* .supports_buft           = */ ggml_backend_metal_supports_buft,
-    /* .offload_op              = */ NULL,
+    /* .offload_op              = */ ggml_backend_metal_offload_op,
     /* .event_new               = */ NULL,
     /* .event_free              = */ NULL,
     /* .event_record            = */ NULL,

I get the following schedule:

## SPLIT #7: Metal # 1 inputs: [ffn_out-0 (   8M)] 
node # 32 (       ADD):              l_out-0 (   8M) [Metal         ]:    Metal#ffn_out-0#0 (   8M) [ NULL         ]            ffn_inp-0 (   8M) [Metal         ]
node # 33 (  RMS_NORM):               norm-1 (   8M) [Metal         ]:              l_out-0 (   8M) [Metal         ]

## SPLIT #8: Metal # 1 inputs: [blk.1.attn_norm.weight (  16K)] 
node # 34 (       MUL):          attn_norm-1 (   8M) [Metal         ]:               norm-1 (   8M) [Metal         ] Metal#blk.1.attn_nor (  16K) [ NULL         ]

## SPLIT #9: CPU # 0 inputs: 
node # 35 (   MUL_MAT):               Qcur-1 (   8M) [  CPU         ]:  blk.1.attn_q.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [Metal         ]
node # 37 (      ROPE):               Qcur-1 (   8M) [  CPU         ]:    Qcur-1 (reshaped) (   8M) [  CPU         ]              inp_pos (   2K) [  CPU         ]
node # 38 (   MUL_MAT):               Kcur-1 (   8M) [  CPU         ]:  blk.1.attn_k.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [Metal         ]
node # 40 (      ROPE):               Kcur-1 (   8M) [  CPU         ]:    Kcur-1 (reshaped) (   8M) [  CPU         ]              inp_pos (   2K) [  CPU         ]
node # 41 (   MUL_MAT):               Vcur-1 (   8M) [  CPU         ]:  blk.1.attn_v.weight (  17M) [  CPU         ]          attn_norm-1 (   8M) [Metal         ]
node # 43 (       CPY): k_cache_view-1 (copy (   4M) [  CPU         ]:               Kcur-1 (   8M) [  CPU         ]       k_cache_view-1 (   4M) [  CPU         ]
node # 46 (       CPY): v_cache_view-1 (copy (   4M) [  CPU         ]:  Vcur-1 (transposed) (   8M) [  CPU         ]       v_cache_view-1 (   4M) [  CPU         ]
node # 50 (   MUL_MAT):                 kq-1 (  32M) [  CPU         ]:                  k-1 (   4M) [  CPU         ]                  q-1 (   8M) [  CPU         ]
node # 51 (  SOFT_MAX):    kq_soft_max_ext-1 (  32M) [  CPU         ]:                 kq-1 (  32M) [  CPU         ]              KQ_mask (   1M) [  CPU         ]
node # 52 (   MUL_MAT):                kqv-1 (   8M) [  CPU         ]:                  v-1 (   4M) [  CPU         ]    kq_soft_max_ext-1 (  32M) [  CPU         ]
node # 54 (      CONT):    kqv_merged_cont-1 (   8M) [  CPU         ]:         kqv_merged-1 (   8M) [  CPU         ]
node # 55 (   MUL_MAT):            kqv_out-1 (   8M) [  CPU         ]: blk.1.attn_output.we (  17M) [  CPU         ]    kqv_merged_cont-1 (   8M) [  CPU         ]

## SPLIT #10: Metal # 1 inputs: [kqv_out-1 (   8M)] 
node # 56 (       ADD):            ffn_inp-1 (   8M) [Metal         ]:    Metal#kqv_out-1#0 (   8M) [ NULL         ]              l_out-0 (   8M) [Metal         ]
node # 57 (  RMS_NORM):               norm-1 (   8M) [Metal         ]:            ffn_inp-1 (   8M) [Metal         ]

## SPLIT #11: Metal # 1 inputs: [blk.1.ffn_norm.weight (  16K)] 
node # 58 (       MUL):           ffn_norm-1 (   8M) [Metal         ]:               norm-1 (   8M) [Metal         ] Metal#blk.1.ffn_norm (  16K) [ NULL         ]

## SPLIT #12: CPU # 0 inputs: 
node # 59 (   MUL_MAT):           ffn_gate-1 (  21M) [  CPU         ]: blk.1.ffn_gate.weigh (  45M) [  CPU         ]           ffn_norm-1 (   8M) [Metal         ]

How does the logic decide to also offload nodes #56 (ADD) and #57 (RMS_NORM) in addition to #58 (MUL)?

slaren · 2024-06-12T10:14:52Z

In the first pass, ops with weights are assigned the backend of the weight. offload_op is used at this point to allow overriding this assignment when the batch size is large enough that it may be worth to copy the weight to VRAM. Then these initial assignments are expanded to the rest of the ops. In this case, what is likely happening is that #58 is assigned to Metal due to offload_op, and then this assignment was expanded to the adjacent ops. You can enable the GET_CAUSE/SET_CAUSE macros to find exactly at which step the assignment was made.

This will cause the weight to be copied to a backend that supports the op, which is very costly. The weight should have been stored in a buffer of a backend that can run the op, but llama.cpp cannot do this automatically at the moment. ggml-ci

* move BLAS to a separate backend * rename GGML_USE_OPENBLAS to GGML_USE_BLAS * alloc : reuse same buffer when the same buffer type if used multiple times * set number of threads automatically for openblas and blis * sched : print assignments when GGML_SCHED_DEBUG env variable is set * sched : allow ops with weights on an incompatible buffer type This will cause the weight to be copied to a backend that supports the op, which is very costly. The weight should have been stored in a buffer of a backend that can run the op, but llama.cpp cannot do this automatically at the moment. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

slaren mentioned this pull request Apr 20, 2024

ggml:add new member in GGML's internal data structure ggml-org/whisper.cpp#2073

Closed

mofosyne added Review Complexity : High Generally require indepth knowledge of LLMs or GPUs refactoring Refactoring labels May 10, 2024

slaren mentioned this pull request May 30, 2024

ggml-backend: refine backend subsystem for CPU&GPU / CPU&NPU mixed inference more easily for a specified GGML backend #7641

Closed

This comment was marked as off-topic.

Sign in to view

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label May 30, 2024

slaren mentioned this pull request Jun 3, 2024

ggml: Support OpenMP for multi-thread processing #7606

Merged

slaren force-pushed the sl/blas-backend branch from d7cc6bc to cde46d2 Compare June 4, 2024 00:28

github-actions Bot added the build Compilation issues label Jun 4, 2024

slaren force-pushed the sl/blas-backend branch 3 times, most recently from 2b5c73d to ca91205 Compare June 4, 2024 23:16

github-actions Bot added Vulkan Issues specific to the Vulkan backend SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language Kompute https://github.com/KomputeProject/kompute/ labels Jun 4, 2024

slaren added 4 commits June 5, 2024 15:16

move BLAS to a separate backend

7f58793

rename GGML_USE_OPENBLAS to GGML_USE_BLAS

b88957e

add support for out_prod

77f88e3

alloc : reuse same buffer when the same buffer type if used multiple …

845fa20

…times

slaren force-pushed the sl/blas-backend branch from ca91205 to 845fa20 Compare June 6, 2024 00:18

support multithreaded dequantization with std::async when openmp is n…

2bfdb7f

…ot available

fix apple build

0425305

slaren marked this pull request as ready for review June 6, 2024 23:58

slaren added 2 commits June 7, 2024 02:58

ggml-ci

2dd049e

reuse main thread

63e06d0

ggerganov self-requested a review June 12, 2024 07:17

ggerganov reviewed Jun 12, 2024

View reviewed changes

Comment thread ggml-blas.cpp Outdated

Comment thread ggml-blas.cpp Outdated

Apply suggestions from code review

a54b791

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

This comment was marked as off-topic.

Sign in to view

ggerganov approved these changes Jun 12, 2024

View reviewed changes

slaren mentioned this pull request Jun 12, 2024

[experimental]backend: add new oneDNN backend ggml-org/ggml#855

Draft

slaren added 2 commits June 13, 2024 02:19

fix metal being used in layers not offloaded

ae9cd85

slaren merged commit f578b86 into master Jun 13, 2024

slaren deleted the sl/blas-backend branch June 13, 2024 01:11

stduhpf mentioned this pull request Jun 17, 2024

Bug: Vulkan, I-quants partially working since PR #6210 (very slow, only with all repeating layers offloaded) #7976

Closed

ggerganov mentioned this pull request Jun 18, 2024

Add Intel Advanced Matrix Extensions (AMX) support to ggml #7707

Closed

ggerganov mentioned this pull request Jul 2, 2024

Inference support for T5 and FLAN-T5 model families #8141

Merged

4 tasks

slaren mentioned this pull request Aug 18, 2024

Bug: MiniCPM-V-2.6 commit d565bb2fd5a2a58b9924a7a34e77a87c78c52137 causing crash in moondream #9066

Closed

Uh oh!

Conversation

slaren commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Mar 21, 2024

Uh oh!

github-actions Bot commented May 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

slaren commented May 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slaren commented Jun 6, 2024

Uh oh!

ggerganov commented Jun 6, 2024

Uh oh!

slaren commented Jun 11, 2024

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

slaren commented Jun 12, 2024

Uh oh!

slaren commented Jun 12, 2024

Uh oh!

slaren commented Jun 12, 2024

Uh oh!

This comment was marked as off-topic.

ggerganov commented Jun 12, 2024

Uh oh!

zhouwg commented Jun 12, 2024

Uh oh!

ggerganov commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slaren commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

slaren commented Mar 21, 2024 •

edited

Loading

github-actions Bot commented May 11, 2024 •

edited

Loading

slaren commented May 30, 2024 •

edited

Loading

ggerganov commented Jun 12, 2024 •

edited

Loading

slaren commented Jun 12, 2024 •

edited

Loading