Add EXAONE MoE implementations

[nuxlear]

Make sure to read the contributing guidelines before submitting a PR

Add EXAONE-MoE architecture for the K-EXAONE model released by LG AI Research.

This PR adds the modeling code of EXAONE-MoE (K-EXAONE), and its chat template with according parsers, which is available at the fork of the LG AI Research:
https://github.com/Aim-Highest/llama.cpp

[pwilkin]

Do we really need a new chat format here? Seems like the standard tool calling format with JSON and <tool_call> tags, besides the optional markdown inside. Is that markdown a real thing?

[nuxlear]

@pwilkin Could you clarify which part you’re referring to?

What I intended here was to combine the reasoning parser from deepseek_v3_1 and the tool-calling format from hermes_2_pro, since K-EXAONE needs both. I could not find use this combination without additional implementation.

[nuxlear]

@CISC I’m working on implementing the MTP logic (it seems to be preserved but not actually used yet, right?).
One difference from other models is that exaone-moe reuses the base model’s token_embd and output layers, instead of defining its own weights.

Is there a recommended or proper way to load and use the base model’s token_embd and output tensors within load_tensors()?
Thank you very much for all your help!

[pwilkin]

@pwilkin Could you clarify which part you’re referring to?

What I intended here was to combine the reasoning parser from deepseek_v3_1 and the tool-calling format from hermes_2_pro, since K-EXAONE needs both. I could not find use this combination without additional implementation.

Okay, it might be that you're right (I was sure there was another model with standard Hermes toolcalling + reasoning, but maybe not).

[nuxlear]

I tested your suggestion encountered the following errors:

1. when using

layer.nextn.embed_tokens     = create_tensor(tn(LLM_TENSOR_NEXTN_EMBED_TOKENS,     "weight"), {n_embd, n_vocab}, flags | TENSOR_NOT_REQUIRED);
layer.nextn.shared_head_head = create_tensor(tn(LLM_TENSOR_NEXTN_SHARED_HEAD_HEAD, "weight"), {n_embd, n_vocab}, flags | TENSOR_NOT_REQUIRED);

the error log is as below:

Loading model... //weka/jwhwang/llama.cpp/ggml/src/ggml-backend.cpp:194: GGML_ASSERT(buffer) failed
-/weka/jwhwang/llama.cpp/build/bin/libggml-base.so.0(+0x53aac)[0x751919802aac]
/weka/jwhwang/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x281)[0x751919802d66]
/weka/jwhwang/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x173)[0x751919802f60]
/weka/jwhwang/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_buffer_get_type+0x44)[0x75191981e836]
/weka/jwhwang/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_buffer_is_host+0x1c)[0x75191981e726]
/weka/jwhwang/llama.cpp/build/bin/libllama.so.0(_ZN23llm_graph_input_out_ids9set_inputEPK12llama_ubatch+0x6f)[0x75191d54dd01]
/weka/jwhwang/llama.cpp/build/bin/libllama.so.0(_ZN16llm_graph_result10set_inputsEPK12llama_ubatch+0x83)[0x75191d5504a9]
/weka/jwhwang/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x2f2)[0x75191d4ea96e]
/weka/jwhwang/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x80e)[0x75191d4ebf2a]
/weka/jwhwang/llama.cpp/build/bin/libllama.so.0(llama_decode+0x20)[0x75191d4f305f]
./build/bin/llama-cli(+0x3936f0)[0x5f4dd28cf6f0]
./build/bin/llama-cli(+0x1ca925)[0x5f4dd2706925]
./build/bin/llama-cli(+0x1a44b6)[0x5f4dd26e04b6]
./build/bin/llama-cli(+0xa7753)[0x5f4dd25e3753]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7519192321ca]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x75191923228b]
./build/bin/llama-cli(+0xa7195)[0x5f4dd25e3195]
\Aborted (core dumped)

2. when using

layer.nextn.embed_tokens     = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, flags | TENSOR_NOT_REQUIRED);
layer.nextn.shared_head_head = create_tensor(tn(LLM_TENSOR_OUTPUT,     "weight"), {n_embd, n_vocab}, flags | TENSOR_NOT_REQUIRED);

the error log is as below:

Loading model... \llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 781, got 783
llama_model_load_from_file_impl: failed to load model
|llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
-llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 781, got 783
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'K-EXAONE-236B-A23B-BF16.gguf'
srv    load_model: failed to load model, 'K-EXAONE-236B-A23B-BF16.gguf'

Failed to load the model

Is there any point I'm missing?

[pwilkin]

@nuxlear haven't checked the entire code, but you might've forgotten to add the default ops for those tensor types to LLM_TENSOR_INFOS? Then the scheduler doesn't know where to put them.

[nuxlear]

@pwilkin Thank you for your attention.

It seems they are already added to LLM_TENSOR_INFOS, I think.

static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
    ...
    // NextN/MTP tensors are currently ignored (reserved for future MTP support)
    // These tensors only exist in the last layer(s) and are treated as output tensors
    {LLM_TENSOR_NEXTN_EH_PROJ,              {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
    {LLM_TENSOR_NEXTN_EMBED_TOKENS,         {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_GET_ROWS}},
    {LLM_TENSOR_NEXTN_ENORM,                {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_GET_ROWS}},
    {LLM_TENSOR_NEXTN_HNORM,                {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL}},
    {LLM_TENSOR_NEXTN_SHARED_HEAD_HEAD,     {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
    {LLM_TENSOR_NEXTN_SHARED_HEAD_NORM,     {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL}},
};

[pwilkin]

Ah, sorry, I missed that you're double-tapping those tensors. In this case, you want to use TENSOR_DUPLICATED.

[pwilkin]

Here's some sample snippet for a similar case from existing code:

                    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);

                    // output
                    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
                    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, TENSOR_NOT_REQUIRED);

                    // if output is NULL, init from the input tok embed
                    if (output == NULL) {
                        output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
                    }

[nuxlear]

Although I use the code below, the GGML_ASSERT(buffer) error raised.

layer.nextn.embed_tokens     = create_tensor(tn(LLM_TENSOR_NEXTN_EMBED_TOKENS,     "weight", i), {n_embd, n_vocab}, flags | TENSOR_NOT_REQUIRED);
if (layer.nextn.embed_tokens == NULL) {
    layer.nextn.embed_tokens = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
}
layer.nextn.shared_head_head = create_tensor(tn(LLM_TENSOR_NEXTN_SHARED_HEAD_HEAD, "weight", i), {n_embd, n_vocab}, flags | TENSOR_NOT_REQUIRED);
if (layer.nextn.shared_head_head == NULL) {
    layer.nextn.shared_head_head = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
}

[CISC]

Ok, I think we can merge after CI, can someone do a last verification that conversion and inference is working in the meantime?

[nicoboss]

Ok, I think we can merge after CI, can someone do a last verification that conversion and inference is working in the meantime?

Convert unfortunately fails with the following error on latest master:

venv/bin/python convert_hf_to_gguf.py /bpool/K-EXAONE-236B-A23B --outfile=/mradermacher/tmp/quant/K-EXAONE-236B-A23B
(...)
INFO:hf-to-gguf:blk.47.ffn_up_shexp.weight,           torch.bfloat16 --> BF16, shape = {6144, 2048}
INFO:hf-to-gguf:blk.47.ffn_norm.weight,               torch.bfloat16 --> F32, shape = {6144}
INFO:hf-to-gguf:output_norm.weight,                   torch.bfloat16 --> F32, shape = {6144}
Traceback (most recent call last):
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 11437, in <module>
    main()
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 11431, in main
    model_instance.write()
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 684, in write
    self.prepare_tensors()
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 8846, in prepare_tensors
    super().prepare_tensors()
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 555, in prepare_tensors
    for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 8843, in modify_tensors
    return [(self.map_tensor_name(name), data_torch)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 510, in map_tensor_name
    raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.layers.48.input_layernorm.weight'

[CISC]

  File "/apool/llama.cpp/convert_hf_to_gguf.py", line 510, in map_tensor_name
    raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.layers.48.input_layernorm.weight'

Hmmm, that sounds strange, it should not be a problem...

[nuxlear]

The problem is that the HF checkpoint (https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B) includes MTP weights, while config.json does not have num_nextn_predict_layer, so the block count remains unchanged, which leads to a mapping error.

Two options are available:

1. simple add "num_nextn_predict_layers": 1 to your HF config.json, or
2. update the default value of num_nextn_predict_layers to 1 in convert_hf_to_gguf.py

I think option 2 is better.

[CISC]

The problem is that the HF checkpoint (https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B) includes MTP weights, while config.json does not have num_nextn_predict_layer, so the block count remains unchanged, which leads to a mapping error.

Two options are available:

1. simple add "num_nextn_predict_layers": 1 to your HF config.json, or
2. update the default value of num_nextn_predict_layers to 1 in convert_hf_to_gguf.py

I think option 2 is better.

Or, you know, update your config: :)

llama.cpp/convert_hf_to_gguf.py

Lines 8777 to 8778 in 60591f0

	# This is because HF config of K-EXAONE does not have `num_nextn_predict_layers` at now.
	# Will be updated when HF config is updated.

[nuxlear]

Yes, there is a third option, however, the mismatch between

llama.cpp/convert_hf_to_gguf.py

Line 8760 in 60591f0

self.block_count = self.hparams["num_hidden_layers"] + self.hparams.get("num_nextn_predict_layers", 0)

and

llama.cpp/convert_hf_to_gguf.py

Lines 8775 to 8779 in 60591f0

	# For here, we hard-code the number of NextN/MTP layers to 1 for K-EXAONE,
	# so that we can convert MTP weights to GGUF format for speculative decoding.
	# This is because HF config of K-EXAONE does not have `num_nextn_predict_layers` at now.
	# Will be updated when HF config is updated.
	self.gguf_writer.add_nextn_predict_layers(self.hparams.get("num_nextn_predict_layers", 1))

still remains, so one of them should be fixed.

Regardless of this, we will add "num_nextn_predict_layers": 1 to our config anyway :)

[CISC]

Yes, there is a third option, however, the mismatch still remains, so one of them should be fixed.

Yes, definitely, please submit a PR, I suggest setting default to 0 as the safer solution.

[nicoboss]

After adding "num_nextn_predict_layers": 1 to config.json convert worked without any issues but I can't run the converted model. I tried booth using the CUDA and the CPU backend from latest master but got the following error:

root@AI:/apool/llama.cpp/build/bin# ./llama-cli -m /mradermacher/tmp/quant/K-EXAONE-236B-A23B.gguf -v
(...)
llama_kv_cache: layer  46: dev = CPU
llama_kv_cache: layer  47: filtered
llama_kv_cache: layer  48: dev = CPU
llama_kv_cache:        CPU KV buffer size =     0.00 MiB
llama_kv_cache: size =  111.00 MiB (   768 cells,  37 layers,  1/1 seqs), K (f16):   55.50 MiB, V (f16):   55.50 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 1
llama_context: max_nodes = 6128
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
/apool/llama.cpp/src/llama-graph.cpp:1052: fatal error
[New LWP 654520]
[New LWP 654521]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000073888a262c17 in __GI___wait4 (pid=654522, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0  0x000073888a262c17 in __GI___wait4 (pid=654522, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x000073888a6a36db in ggml_print_backtrace () from libggml-base.so.0
#2  0x000073888a6a382e in ggml_abort () from libggml-base.so.0
#3  0x000073888a81b935 in llm_graph_context::build_moe_ffn(ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, long, long, llm_ffn_op_type, bool, bool, float, llama_expert_gating_func_type, int, ggml_tensor*) const () from libllama.so.0
#4  0x000073888a81c494 in llm_graph_context::build_moe_ffn(ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, long, long, llm_ffn_op_type, bool, bool, float, llama_expert_gating_func_type, int, ggml_tensor*) const () from libllama.so.0
#5  0x000073888a90fb2b in llm_build_exaone_moe::llm_build_exaone_moe(llama_model const&, llm_graph_params const&) () from libllama.so.0
#6  0x000073888a857472 in llama_model::build_graph(llm_graph_params const&) const () from libllama.so.0
#7  0x000073888a7e8e2b in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from libllama.so.0
#8  0x000073888a7eeed9 in llama_context::llama_context(llama_model const&, llama_context_params) () from libllama.so.0
#9  0x000073888a7ef47d in llama_init_from_model () from libllama.so.0
#10 0x000073888a7c243a in llama_get_device_memory_data(char const*, llama_model_params const*, llama_context_params const*, std::vector<ggml_backend_device*, std::allocator<ggml_backend_device*> >&, unsigned int&, unsigned int&, unsigned int&, ggml_log_level) () from libllama.so.0
#11 0x000073888a7c2f47 in llama_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from libllama.so.0
#12 0x000073888a7c636e in llama_params_fit () from libllama.so.0
#13 0x00005c3d4d289826 in common_init_result::common_init_result(common_params&) ()
#14 0x00005c3d4d28b059 in common_init_from_params(common_params&) ()
#15 0x00005c3d4d174c75 in server_context_impl::load_model(common_params const&) ()
#16 0x00005c3d4d0c55e6 in main ()
/[Inferior 1 (process 654517) detached]
Aborted

[CISC]

After adding "num_nextn_predict_layers": 1 to config.json convert worked without any issues but I can't run the converted model. I tried booth using the CUDA and the CPU backend from latest master but got the following error:

/apool/llama.cpp/src/llama-graph.cpp:1052: fatal error

OK, my bad, I didn't notice that scoring_func was removed from the config (why?):
https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B/commit/27c3a76f02c76138b854cae2d14850eadb4c86db#d2h-025836

And I forgot to ensure that this wasn't false:

llama.cpp/src/llama-model.cpp

Line 1954 in 60591f0

ml.get_key(LLM_KV_EXPERT_GATING_FUNC, hparams.expert_gating_func, false);

@nuxlear Can you please restore scoring_func in the config again?

[CISC]

@nicoboss Merged fix, requires updated config.json with scoring_func.

[nicoboss]

Thank you all so much for quickly fixing this issue. I can confirm that after getting latest master, latest model and adding "scoring_func": "sigmoid" to the config.json it successfully converted and gives great responses and is incredibly fast for its size. The only thing I noticed is that while loading verbose the model the following warning about unused tensors appear but it doesn't seem to matter:

create_tensor: loading tensor blk.47.ffn_up_shexp.weight
model has unused tensor blk.48.attn_q.weight (size = 41287680 bytes) -- ignoring
model has unused tensor blk.48.attn_k.weight (size = 5160960 bytes) -- ignoring
model has unused tensor blk.48.attn_v.weight (size = 5160960 bytes) -- ignoring
model has unused tensor blk.48.attn_output.weight (size = 41287680 bytes) -- ignoring
model has unused tensor blk.48.attn_norm.weight (size = 24576 bytes) -- ignoring
model has unused tensor blk.48.attn_q_norm.weight (size = 512 bytes) -- ignoring
model has unused tensor blk.48.attn_k_norm.weight (size = 512 bytes) -- ignoring
model has unused tensor blk.48.ffn_norm.weight (size = 24576 bytes) -- ignoring
model has unused tensor blk.48.ffn_gate.weight (size = 92897280 bytes) -- ignoring
model has unused tensor blk.48.ffn_down.weight (size = 92897280 bytes) -- ignoring
model has unused tensor blk.48.ffn_up.weight (size = 92897280 bytes) -- ignoring
model has unused tensor blk.48.nextn.eh_proj.weight (size = 61931520 bytes) -- ignoring
model has unused tensor blk.48.nextn.enorm.weight (size = 24576 bytes) -- ignoring
model has unused tensor blk.48.nextn.hnorm.weight (size = 24576 bytes) -- ignoring
model has unused tensor blk.48.nextn.shared_head_norm.weight (size = 24576 bytes) -- ignoring
load_tensors: tensor 'token_embd.weight' (q6_K) (and 0 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead

[CISC]

The only thing I noticed is that while loading verbose the model the following warning about unused tensors appear but it doesn't seem to matter:

This is normal.

[nuxlear]

@CISC @nicoboss Sorry for the delay. We've added "scoring_func": "sigmoid" to our config.json.