[BUG]: Linux-arm64 crashes with a TypeInitializationException

[BartjeD]

Description

When attempting to run a LLamasharp API on linux-arm64, compiled als a self-contained runtime: I get a runtime error for type initialization, which says the CPU library is missing.

But I'm very sure it's installed. And of course the API works locally on Windows 11, without using a GPU.

Exception:
System.TypeInitializationException: The type initializer for 'LLama.Native.NativeApi' threw an exception. ---> LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded. It could be one of the following reasons: 1. No LLamaSharp backend was installed. Please search LLamaSharp.Backend and install one of them. 2. You are using a device with only CPU but installed cuda backend. Please install cpu backend instead. 3. One of the dependency of the native library is missed

I checked the release build files, for the given API.
To be sure that the cpu lib ggml library is present, which it seems to be:

Is ARM-64 supported? Do I need to install any libraries on the hosting environment?

Reproduction Steps

Run CPU only instance on linux-arm64, it will fail with LLama runtime error: `System.TypeInitializationException.... see above.

Environment & Configuration

• Operating system: Debian 12
• .NET runtime version: .net 9, 9.0.101
• LLamaSharp version: 0.24.0
• CUDA version (if you are using cuda backend): n/a
• CPU & GPU device: arm64 Ampere Altra

Known Workarounds

N/a

[BartjeD]

I'm assuming it's supported, since these changes were comitted: #1137 and: #1138

[martindevans]

Arm64 should be supported, althougn it's qutie new and not tested in CI yet so I expect some bugs there!

Does it work if you run it from VS (or with dotnet run) instead of an actual build? I ask because there's been a known issue for a while with self-contained builds placing the native runtime files in the wrong directory (it flattens the folder hierarchy for some reason) - that moans the loading system can't find the DLLs it's expecting.

[BartjeD]

Should I put the files 's in the same folder they end up in, in the debug build?
I can give that a try on the server

/runtimes/linux-arm64/native/

[BartjeD]

Still gives the same exception:

[martindevans]

That looks like the right path. If you add this to the start of your program (before any other LLamaSharp calls) you can get verbose logging info which should show the exact paths it's looking for:

NativeLibraryConfig
   .All
   .WithLogCallback((level, message) => Console.WriteLine($"LL# [{level}]: {message}");

[sangyuxiaowu]

Aha, I have experience with this. When using the runtimes path, I need to be on the Windows platform. My Jetson Agx arm64 device requires me to compile the CUDA version myself and copy the SO file to the same directory as the program.
You can try compiling llama.cpp yourself on your arm64 device.

[BartjeD]

Oke 1 step further now:

The file is found, but cannot be loaded

[martindevans]

Could you try running ldd on the file (linux-arm64/native/libllama.so), that'll tell us if there's some dependency missing.

[sangyuxiaowu]

I seem to have found the problem. Need to upgrade glibc.

My device can run normally as I compiled it myself, but there are indeed issues with the Nuget service, which can be used as a reference:

My compiled Cuda

agx@ubuntu:/work/gpt$ ldd libllama.so
        linux-vdso.so.1 (0x0000ffffb7886000)
        libggml.so (0x0000ffffb76b0000)
        libggml-base.so (0x0000ffffb75d0000)
        libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffffb7390000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffb72f0000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffffb72c0000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffb7110000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffffb784d000)
        libggml-cpu.so (0x0000ffffb7040000)
        libggml-cuda.so (0x0000ffffb3fa0000)
        libgomp.so.1 => /lib/aarch64-linux-gnu/libgomp.so.1 (0x0000ffffb3f40000)
        libcudart.so.12 => /usr/local/cuda/lib64/libcudart.so.12 (0x0000ffffb3e70000)
        libcublas.so.12 => /usr/local/cuda/lib64/libcublas.so.12 (0x0000ffffac9a0000)
        libcuda.so.1 => /usr/local/cuda/compat/libcuda.so.1 (0x0000ffffaad00000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffffaace0000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffffaacc0000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000ffffaaca0000)
        libcublasLt.so.12 => /usr/local/cuda/lib64/libcublasLt.so.12 (0x0000ffff8c530000)
        libnvrm_gpu.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so (0x0000ffff8c4c0000)
        libnvrm_mem.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so (0x0000ffff8c4a0000)
        libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000ffff8c470000)
        libnvsocsys.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsocsys.so (0x0000ffff8c450000)
        libnvrm_sync.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_sync.so (0x0000ffff8c430000)
        libnvsciipc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so (0x0000ffff8c400000)
        libnvrm_chip.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so (0x0000ffff8c3e0000)
        libnvrm_host1x.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_host1x.so (0x0000ffff8c3b0000)

Nuget

agx@ubuntu:/work/gpt/so$ ldd libllama.so
./libllama.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./libllama.so)
./libllama.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by libggml-base.so)
        linux-vdso.so.1 (0x0000ffff8d1f6000)
        libggml.so (0x0000ffff8d000000)
        libggml-base.so (0x0000ffff8cf20000)
        libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff8cce0000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff8cc40000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff8cc10000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff8ca60000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff8d1bd000)
        libggml-cpu.so (0x0000ffff8c980000)
        libgomp.so.1 => /lib/aarch64-linux-gnu/libgomp.so.1 (0x0000ffff8c920000)

[sangyuxiaowu]

This situation is indeed quite common in embedded systems, as embedded devices typically run relatively older, stable versions of operating systems. So, it is recommended to compile it yourself. Upgrading glibc may pose greater risks.

[BartjeD]

Yes:
version `GLIBC_2.38' not found (required by libggml-base.so)
linux-vdso.so.1 (0x0000ffff9a390000)
libggml.so (0x0000ffff9a190000)
libggml-base.so (0x0000ffff9a0b0000)
libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff99e90000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff99df0000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff99db0000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff99c00000)
/lib/ld-linux-aarch64.so.1 (0x0000ffff9a353000)
libggml-cpu.so (0x0000ffff99b20000)
libgomp.so.1 => not found

[sangyuxiaowu]

If you don't want to compile, you can also try the CUDA version I compiled first. It should return to the CPU by itself, maybe it can be used.
Try using the SO file in the compressed file.
LLamaWorker-cuda12.2-linux-arm64-b5415.zip

[BartjeD]

If I switch to Ubuntu 24.04 LTS it will have GLIBC_2.39, which I think will be OK, we'll find out

[BartjeD]

Ok, now on Ubuntu 24.04,

it says libgomp.so.1 => not found
The glibc error is resolved.

The exception is still the same: System.TypeInitializationException: The type initializer for 'LLama.Native.NativeApi' threw an exception. ---> LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded