Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 56 additions & 9 deletions docs/content/features/GPU-acceleration.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,58 @@ Due to the nature of ROCm it is best to run all implementations in containers as
- Make sure to do not use GPU assigned for compute for desktop rendering.
- Ensure at least 100GB of free space on disk hosting container runtime and storing images prior to installation.

### AMD Strix Halo / gfx1151 (RDNA 3.5)

AMD Ryzen AI MAX+ (Strix Halo) APUs with an integrated Radeon 8060S (gfx1151 / RDNA 3.5) are
supported with ROCm 7.11.0+. These systems provide up to 96 GB of unified VRAM accessible by the GPU.

Tested on: Geekom A9 Mega (AMD Ryzen AI MAX+ 395, ROCm 7.11.0, Ubuntu 24.04, kernel 6.14).

**Required kernel boot parameters** (add to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`, then run `update-grub`):
```
iommu=pt amdgpu.gttsize=126976 ttm.pages_limit=32505856
```

**Required environment variables** for gfx1151 (set automatically in the ROCm/hipblas image):

| Variable | Value | Purpose |
|----------|-------|---------|
| `HSA_OVERRIDE_GFX_VERSION` | `11.5.1` | Tells the HSA runtime to use gfx1151 code objects |
| `ROCBLAS_USE_HIPBLASLT` | `1` | Prefer hipBLASLt over rocBLAS for GEMM (required for gfx1151) |
| `HSA_XNACK` | `1` | Enable XNACK (memory-fault retry) for APU unified memory |
| `HSA_ENABLE_SDMA` | `0` | Disable SDMA engine — causes hangs on APU/iGPU configs |

> **Warning:** Do **not** set `GGML_CUDA_ENABLE_UNIFIED_MEMORY`. The C-level check is
> `getenv(...) != nullptr`, so even `=0` activates `hipMallocManaged` (allocates from
> system RAM instead of the 96 GB VRAM pool).

**Running LocalAI on gfx1151** (use the standard ROCm/hipblas image — there is only one ROCm image, and it ships with ROCm 7.x by default):
```yaml
image: quay.io/go-skynet/local-ai:master-gpu-hipblas
environment:
- HSA_OVERRIDE_GFX_VERSION=11.5.1
- ROCBLAS_USE_HIPBLASLT=1
- HSA_XNACK=1
- HSA_ENABLE_SDMA=0
devices:
- /dev/dri
- /dev/kfd
group_add:
- video
```

> **Note:** When updating the image, always recreate the container (`docker compose up --force-recreate`)
> rather than just restarting it. `docker compose restart` preserves the old container environment
> and will not pick up updated env vars from the image.

For llama.cpp models, enable flash attention (`--flash-attention`) and disable mmap (`--no-mmap`) for best performance on APU systems.

### Limitations

Ongoing verification testing of ROCm compatibility with integrated backends.
Please note the following list of verified backends and devices.

LocalAI hipblas images are built against the following targets: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1102, gfx1200, gfx1201
LocalAI hipblas images are built against the following targets: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1102, gfx1151, gfx1200, gfx1201

**Note:** Starting with ROCm 6.4, AMD removed rocBLAS kernel support for older architectures (gfx803, gfx900, gfx906). Since llama.cpp and other backends depend on rocBLAS for matrix operations, these GPUs (e.g. Radeon VII) are no longer supported in pre-built images.

Expand All @@ -181,14 +227,15 @@ If your device is not one of the above targets, you must specify the correspondi

The devices in the following list have been tested with `hipblas` images.

| Backend | Verified | Devices |
| ---- | ---- | ---- |
| llama.cpp | yes | MI100 (gfx908), MI210/250 (gfx90a) |
| diffusers | yes | MI100 (gfx908), MI210/250 (gfx90a) |
| whisper | no | none |
| coqui | no | none |
| transformers | no | none |
| vllm | no | none |
| Backend | Verified | Devices | ROCm Version |
| ---- | ---- | ---- | ---- |
| llama.cpp | yes | MI100 (gfx908), MI210/250 (gfx90a) | 7.x |
| llama.cpp | yes | Radeon 8060S / gfx1151 (Strix Halo) | 7.11.0 |
| diffusers | yes | MI100 (gfx908), MI210/250 (gfx90a) | 7.x |
| whisper | no | none | - |
| coqui | no | none | - |
| transformers | no | none | - |
| vllm | no | none | - |

**You can help by expanding this list.**

Expand Down
Loading