Feature request: official gfx1151 (Strix Halo / RDNA3.5) backend builds on ROCm 7.14

[walcz-de]

Is your feature request related to a problem? Please describe.

Users on gfx1151 (AMD Ryzen AI Max+ / Strix Halo, RDNA3.5) must build the LocalAI backends (llama-cpp, diffusers, rerankers, whisper) themselves on every ROCm/LocalAI bump. The official backend gallery ships CUDA and some ROCm variants, but no gfx1151 target. Until ROCm 7.13 gfx1151 was preview-only / not in AMD's official catalog, so official builds weren't justifiable. ROCm 7.14 (2026-07-15) makes gfx1151 a first-class, natively-detected target — the historical blocker is gone, and Strix Halo (96–128 GB unified memory, strong MoE decode) is a fast-growing local-inference platform.

Describe the solution you'd like

Add gfx1151 / RDNA3.5 on ROCm 7.14 as an official target in the backend build matrix, published to the gallery like other backends. We run a production gfx1151 stack and are happy to (1) contribute a PR wiring the target into the build workflow using the recipe below, and (2) act as a gfx1151 CI/validation partner — we have the hardware to smoke/perf-test backend releases.

Verified build recipe (production, Ryzen AI Max+ 395):

• Base userspace — Ubuntu 24.04 + two AMD metapackages from the multi-arch repo:

  deb https://repo.amd.com/rocm/packages-multi-arch/ubuntu2404 stable main
  amdrocm7.14-gfx1151            # runtime closure (blas/rand/fft/solver/sparse, ck, dnn,
                                 # rccl/rocshmem, math-common=rocWMMA, amdsmi/hipify/decode/jpeg)
  amdrocm-core-sdk7.14-gfx1151   # build closure (core-dev + developer-tools + rdc + opencl)
  

  Two metapackages pull the complete coherent 7.14 set — no transitive gaps (structurally fixes the librocprofiler-register.so class of missing-lib crashes).
• llama.cpp build flags: -DGGML_HIP=ON -DGGML_HIP_ROCWMMA_FATTN=ON -DAMDGPU_TARGETS=gfx1151 plus -DGGML_HIP_NO_VMM=ON (HIP VMM is unreliable on gfx1151).
• Runtime: HSA_OVERRIDE_GFX_VERSION is not needed on 7.14 (native detection; it was only a 7.12/7.13 workaround). For large single buffers (>~61 GiB) the kernel needs ttm.page_pool_size/ttm.pages_limit raised (the "64 GB cliff", Rocm slow model weight loading past ~64GB mark on Strix Halo (all fine with Vulkan) ggml-org/llama.cpp#15018) — worth a docs note for Strix Halo users running big MoE (we run up to ~83 GB models at a 120 GiB GTT ceiling).
• Shared-lib packaging: derive the runtime lib allowlist dynamically (recursive ldd closure) rather than a static list — transitive ROCm deps shift per version (librocprofiler-register.so, libamd_comgr_loader.so, …).

Describe alternatives you've considered

• Self-building (our current approach): works, but is the single most expensive recurring maintenance item — a custom base image plus atomic llama-cpp/diffusers/rerankers/whisper rebuilds on every ROCm/LocalAI sync.
• Community-only build guide: helps individuals but doesn't give gfx1151 users the "pull from gallery" experience the other targets have.

Additional context

Would this be something you'd want in-tree, and if so, in what shape — a build-matrix entry, a separate ROCm/gfx1151 workflow, or docs + a community build guide? Happy to align on scope first, then open the PR. We can validate on real gfx1151 hardware for each backend release.

[walcz-de]

Update — we ran vLLM on gfx1151 end-to-end and it works as a LocalAI backend. Sharing concrete results in case they help scope this:

Setup: AMD's official rocm/vllm:rocm7.14.0_rdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0 container as the base, LocalAI's backend/python/vllm wrapper on top (gRPC), wired via LOCALAI_EXTERNAL_GRPC_BACKENDS. Ryzen AI Max+ 395 (gfx1151), ROCm 7.14, 128 GB UMA.

What works / what doesn't (gfx1151-specific):

• ✅ compressed-tensors int4 (pack-quantized) MoE loads + serves. This is the reliable 4-bit path.
• ❌ GPTQ: Marlin is CUDA-only → falls back to a Triton w4a16 kernel that asserts (qzeros shape mismatch) and crashes at warmup.
• ⚠️ AITER / QuarkOCP_MX / FP8-MoE: not available on gfx1151 (CDNA-only) — the container logs AITER is not found.
• LocalAI's backend.py uses the V0 AsyncLLMEngine API, which is still importable in vLLM 0.23.1 → no code changes needed.

Performance (Qwen3-30B-A3B, 4-bit, decode aggregate tok/s):

K (concurrency)	1	4	8	16
enforce-eager	27.9	62	105	203
HIP graphs (default)	25.2	56	98	197

• enforce_eager=true is ~5-10% FASTER than HIP graphs on gfx1151 (graph-capture overhead isn't recovered on this fallback-kernel path). Recommend it as the default for RDNA3.5.
• Continuous batching scales ~7.3× (K=1→16), so vLLM is a strong concurrent/batch second engine here (llama.cpp caps earlier), while llama.cpp stays better for single-stream/interactive.

UMA gotcha: default gpu_memory_utilization=0.92 OOM-kills (exit 137) — vLLM's profiler over-reserves KV on unified memory. 0.3–0.5 + enforce_eager is stable and coexists with a llama.cpp instance on the same 120 GiB.

Happy to turn the base-container + wrapper Dockerfile into a PR if a gfx1151/RDNA vLLM backend target is wanted in-tree.

[walcz-de]

Update (2026-07-18): the FROM-scratch backend packaging is the blocker on gfx1151 — not the wheels

Follow-up to the vLLM data above, now with diffusers — same root cause, so this strengthens the case for official gfx1151 builds based on AMD's ROCm container rather than the generic Dockerfile.python (FROM scratch + package-gpu-libs.sh + portable venv) path.

Finding: AMD's torch 2.11.0+rocm7.14.0 (index repo.amd.com/rocm/whl-multi-arch, hip 7.14.60850) fully supports gfx1151 — torch.cuda.get_arch_list() includes gfx1151, and it runs cleanly inside AMD's own container (rocm/vllm:rocm7.14.0_rdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0): a 64² matmul in ~0.05s, a z-image-turbo 1024² image in ~29s. But the same wheel repackaged through Dockerfile.python (portable venv + package-gpu-libs.sh-bundled base ROCm libs) fails on the first GPU op:

torch in the FROM-scratch backend	first GPU op
2.10.0+rocm7.0 (the current requirements-hipblas.txt pin)	segfault (ABI: rocm7.0 torch vs the 7.14 runtime libs of a 7.14 base)
2.11.0+rocm7.14.0 (AMD, gfx1151-capable)	hipErrorInvalidImage even though arch_list has gfx1151 → ROCm-userspace mismatch (comgr / device-libs / HSA) between the bundled/base libs and what the wheel expects

So on gfx1151 the reliable path for torch engines (vLLM and diffusers) is to build FROM AMD's ROCm container and run them as external gRPC backends. Working Dockerfiles (FROM rocm/vllm:rocm7.14.0_rdna…, add only the gRPC layer + for diffusers the PyPI ML stack, torch untouched): happy to share/PR.

diffusers specifics that may be worth upstreaming regardless of the base question:

• av (PyAV) is a hard transitive import (diffusers/pipelines/ltx2/export_utils.py) — missing it makes import diffusers fail.
• On gfx1151 MIOPEN_FIND_MODE=FAST (+ a persistent MIOPEN_USER_DB_PATH) is needed; otherwise MIOpen does a multi-minute exhaustive kernel search on first generate (no pre-tuned gfx1151_*.HIP.fdb.txt), blowing past the gRPC deadline.
• BackendServicer.__init__ doesn't default its attributes (controlnet, img2vid, txt2vid, ltx2_pipeline, compel, clip_skip, pipe, options) — they're only set in LoadModel. If a request reaches the servicer before/without a completed LoadModel (e.g. an external backend restarted while the client still has the model cached), GenerateImage raises 'BackendServicer' object has no attribute 'controlnet'. A small __init__ with defaults fixes it. Happy to PR.

[localai-org-maint-bot]

Cross-linking: there is an open docs PR covering this hardware — #9229 (docs(gpu): add ROCm 7.x and RDNA 3.5 / Strix Halo (gfx1151) to GPU acceleration guide) by @walcz-de. That covers the documentation half of this request; the ask here for official prebuilt gfx1151 backend images would additionally need a gfx1151 entry wired into .github/backend-matrix.yml and the corresponding backend/index.yaml entries.

Labelled area/backends + area/ci accordingly.

[walcz-de]

Validated ROCm 7.14 end-to-end on AMD hardware — and it turns out to be a "TheRock" packaging migration for AMD, not a one-line base bump. Opened #10978 with the fix.

This is platform-wide for AMD, not gfx1151-specific. The base image, package layout and library paths it touches affect every arch in AMDGPU_TARGETS (gfx908/90a/942/1030/1100/1101/1102/1151/1200/1201) — the ROCm/hipblas images don't build (then don't load) on ROCm 7.14 until these are fixed, regardless of the card. gfx1151 / Strix Halo is just the AMD GPU I had to runtime-test on.

gfx1151 is already in AMDGPU_TARGETS and native on 7.14 (no HSA_OVERRIDE_GFX_VERSION); the pin was just too old (rocm/dev-ubuntu-24.04:7.2.1 → :7.14.0-full).

A naive base bump breaks the AMD build in three ways (TheRock guide), all fixed and all architecture-agnostic:

1. Legacy -dev metapackages gone. apt-get install hipblas-dev hipblaslt-dev rocblas-dev → Unable to locate package. Consolidated + arch-split (amdrocm-blas<ver>-gfx*). The -full base ships them, so the install is dropped.
2. ROCm libs scattered, not on the linker path (/opt/rocm/lib, /opt/rocm/lib/rocm_sysdeps/lib, /opt/rocm/llvm/lib, no ld.so.conf.d entry). The ldd-based GPU-lib packaging then silently skips librocm_kpack, librocm_sysdeps_*, rocBLAS, hipBLASLt… and the backend fails to load at runtime on any AMD GPU (cannot open shared object file). CI never catches this. Fixed by registering the three lib dirs before ldconfig.
3. Base bump across the matrix/workflows; base-images.yml rebuilds the gRPC cache on 7.14.

Validated on gfx1151 / Strix Halo: core image + llama-cpp backend build, all ROCm libs bundle, inference runs on the GPU. Since I only have one AMD card, someone with other AMD GPUs (CDNA gfx908/90a/942, RDNA gfx1030/1100/…) should confirm — especially fix #2 (runtime-only).