Add self extend support

[Josh-XT]

Add self extend support

• More info in main : add Self-Extend support ggml-org/llama.cpp#4815

[Josh-XT]

Haven't had time to finish the rest of this, got pulled away in the middle! Sorry - I'll reopen when ready if someone doesn't beat me to the implementation.

[jshph]

Hey @Josh-XT thanks for taking a stab. Curious if the implementation is in the right direction here, I was poking at it yesterday and saw the Params struct being modified. I wonder if that'd affect the parsing of args downstream.

[Josh-XT]

Hey @Josh-XT thanks for taking a stab. Curious if the implementation is in the right direction here, I was poking at it yesterday and saw the Params struct being modified. I wonder if that'd affect the parsing of args downstream.

I was attempting to follow the changes made on the cpp repo but while hanging out with my 4 year old, then he made a mess and I had to stop. 🤣 I was trying to do the cpp server example at the same time when my plans were changed. The implementation is pretty straightforward. I think the grp-attn-n and grp-attn-w could (and probably will) be automated away.

[ExtReMLapin]

Any update on this ?