feat: DRY and XTC samplers

[onzag]

Feature Description

Currently the samplers for top_p temperature, etc... are included by default, but I don't see anywhere, in the docs or in the type definitions ot have DRY and XTC data, which appear clearly in the llama.cpp and doesn't seem like they'd be too hard to implement as they are natively supported.

abetlen/llama-cpp-python#1813

Similar comments about this in llama-cpp-python.

The Solution

Adding the options to dry_sampler and xtc_sampler with the given options for generateCompletition and other functions.

dry_sampler

• enabled boolean
• multiplier number,
• base number,
• allowed_length number
• penalty_last_n number
• seq_breakers: Array

xtc_sampler

• enabled boolean
• probability number
• threshold: number
• min_keep: number
• seed?: number

Considered Alternatives

I couldn't find any workaround, checked grammar defintions, checked for any mention of dry or xtc and couldn't find it.

Additional Context

No response

Related Features to This Feature Request

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

[onzag]

https://github.com/withcatai/node-llama-cpp/blob/master/llama/addon/AddonSampler.cpp

I assume it takes on from here.

[onzag]

I am trying but not a C coder :( nor a NodeJS API native extensions developer.

I think I could figure it out eventually, but it would take me a while to grasp the native side of NApi.

Got trying to get the seq_breakers and num_breakers from what is supposed to be an array list, also the function is defined as

llama_sampler_init_dry(const struct llama_vocab * vocab, int32_t n_ctx_train, float dry_multiplier, float dry_base, int32_t dry_allowed_length, int32_t dry_penalty_last_n, const char** seq_breakers, size_t num_breakers) {

no idea where I'd get the vocab or what that ctx_train is meant to be.

Tho I found this reference llama_sampler_init_dry(vocab, llama_model_n_ctx_train(model), params.dry_multiplier, params.dry_base, params.dry_allowed_length, params.dry_penalty_last_n, c_breakers.data(), c_breakers.size())

if (config.Has("drySampling")) {
        Napi::Value drySamplingValue = config.Get("drySampling");
        if (drySamplingValue.IsObject()) {
            Napi::Object drySampling = drySamplingValue.As<Napi::Object>();
            bool enabled = drySampling.Has("enabled") ? drySampling.Get("enabled").As<Napi::Boolean>().Value() : false;

            if (enabled) {
                auto multiplierDrySampling = drySampling.Has("multiplier")
                    ? drySampling.Get("multiplier").As<Napi::Number>().FloatValue()
                    : 1.0f;
                auto baseDrySampling = drySampling.Has("base")
                    ? drySampling.Get("base").As<Napi::Number>().FloatValue()
                    : 0.0f;
                auto allowedLengthDrySampling = drySampling.Has("allowedLength")
                    ? drySampling.Get("allowedLength").As<Napi::Number>().Int32Value()
                    : 0;
                auto penaltyLastNDrySampling = drySampling.Has("penaltyLastN")
                    ? drySampling.Get("penaltyLastN").As<Napi::Number>().Int32Value()
                    : 0;

                 // TODO
                
            } else if (drySampler != nullptr) {
                freeChain();
                llama_sampler_free(drySampler);
                drySampler = nullptr;
            }
        } else if (drySampler != nullptr) {
            freeChain();
            llama_sampler_free(drySampler);
            drySampler = nullptr;
        }
    } else if (drySampler != nullptr) {
        freeChain();
        llama_sampler_free(drySampler);
        drySampler = nullptr;
    }

[giladgd]

I haven't added support for all samplers that are available in llama.cpp mainly because I wanted to wait and see which ones people actually end up using and stick around, since some of the least used ones were removed over time, and I'd like to avoid breaking changes in node-llama-cpp as much as possible (especially avoid breaking changes that remove functionality).

Regarding the DRY sampler, I think it conflicts with regular repeat penalties, so it'd be important to design the JS API (especially the TS types) to reflect that.
Figuring all the sequence breakers (the seq_breakers array) in advance isn't easy either, and the current implementation in llama.cpp expects you to manually pass them. Theoretically we could designate all the special tokens (control tokens) as sequence breakers, but that wouldn't be enough for most SOTA models, since gpt-oss doesn't have all of its control tokens marked up properly (some popular quantizations but not all of them, and many other models as well) and many models output <think> and </think> (or their custom representation of it) for though segments which are also usually not tagged.

The aim of node-llama-cpp is to provide high-level APIs that work out of the box without requiring deep knowledge into the details how of inference works just to use it with some model, so it'd be important to figure out a good and comprehensive default for seq_breakers (it's might even be better to do that behind the scenes and omit the ability to even configure it) before we introduce support for DRY.

Regarding the XTC sampler, this one looks pretty easy to implement and seems to be widely supported, I can add it this/next week.

[onzag]

@giladgd yes you are right it does, and I guess I didn't think about that, I just saw them in the llama.cpp natively and wondered if they were to be implemented in this since many roleplaying models do perform nicely with these samplers.

I think that one of the main reasons you are not seeing requests for these is probably that few people are actually building apps and doing mostly prototyping or otherwise creating small agents (maybe?), but I am building an electron app for D&D style roleplay and well, it's a bit big, bigger than sillytavern in scope, but in local setups using DRY and XTC is very good for creative writting. That saying, I am also new at this, so figuring out a lot of things still. You can check my rstory repo, but it's a bit of a mess, and it's not currently running, but you get the idea js/local-server/local-llama.js has the barebones implementation so far (needs work still) I have a lot to figure out. But what I mean is that DRY and XTC are very good for creative writting.

I am a big puzzled about sequence breakers are in the DRY implementation, like if this is a developer tool, isn't it totally fine if the developer needs to specify these custom tokens?... like if you penalize these "thinking" tokens because you didn't add these tags in the dry settings and ruin the inference, isn't that just a fine outcome?... like as a developer you should have been aware to add the tokens in the DRY settings, some documentation should be good, like these are advanced settings after all; I mean it works within llama.cpp, I'd think it's acceptable if it needs to be manually specified.

Like I can mess things up equally when setting grammar. Yes One of the best features I have access with this library is grammar and without it just wouldn't be as good, I know it comes from within the underlying llama.cpp but boy it is useful, you can see it everywhere in my repo, it's not in the local server file, I have varying grammar per request depending on requirements.

Personally, of course this is your library and you can disagree, but I see the value of node-llama-cpp in it just works and I want to do inference and focus on the inference part (even with some low level settings); before I was using python and it was just bursting into flames, CUDA crashes, all sort of shenanigans; so many requirements, python llama required babysitting and wasn't portable and I wanted to write the app, I didn't want to deal with writting a websocket server in C I'd rather use python or node; and your lib, works.

In fact your lib was recommended online because of that, because people were frustrated with setting things up and just wanted to write agentic code, that's how I found it.

Of course for me, it's just part of a hobby for D&D like agents, so I may be, thinking differently from most the userbase of this library.

[giladgd]

@onzag I spent a lot of time on making it work so well across all platforms, compute backends and hardware configurations I could, without you needing to manually configure anything. I created this library because I wanted it for myself and encountered the same issues. As a user you just want it to work and usually don't care about all the technical details.

That same approach goes throughout almost all the implementation details.
If some feature or option may be confusing or misleading, then I either want to polish it to match the implementation to your expectations, or not have it at all.

Regarding the DRY sampler, the issue is that to configure the sequence breakers you need to have a lot of technical knowledge and do manual inspection for each model which takes time and manual effort, and most people won't know how to do it and probably won't.
Not configuring it at all (passing an empty list) or misconfiguring it might look like it's working with some models in some scenarios, but building that expectation for users is dangerous since there's a good change that it'll cause issues when you use it in a different scenario or with a different model (not just logical ones, also technical, like the model not stopping to generate output or generating gibberish, etc.).
I have no issue adding advanced features that require advanced knowledge to use, but I don't want to add features that could easily defy user expectations and lead to issues.

BTW I highly recommend you to try scaffolding a project from the node-llama-cpp Electron template, since it comes with a solution to call functions between the renderer and the main process (in a type-safe way without an intermediate definition) that runs node-llama-cpp.

[onzag]

@giladgd thanks, I will check that later, for now I need to restart the process so often that I wanted to keep it separate, at least for now.

I notice that many models have a JSON file with some of such details for some engines at least, I see no mention of DRY nevertheless or something that would appear like it, they appear to be explanations and guidelines on how to create a prompt for the model instead.

Yes that's right I don't think seq_breakers appears anywhere, I found some pieces of code like this...

https://fossies.org/linux/llama.cpp/common/common.h

where it can be seen that...

std::vector<std::string> dry_sequence_breakers = {"\n", ":", "\"", "*"};

Is weirdly what they are doing, which seems suboptimal, but maybe good enough?.... the use of asterisk and quotes within the .cpp code itself appears to showcase that this is somehow the norm for creative writting, as those are exactly the roleplaying style sequence breakers you'd use; and this sampler specifically.

I think that whomever may be thinking of this DRY sampler, may have that exact same purpose because these defaults seem hardcoded.

Of course as you say if I have the "<think"> "" tag, it may be... needed to add the ....

Wait... how about having seq_breakers "defaults", like an object that is like... import SeqBreakersDefaults... and like SeqBreakersDefault.Standard or SeqBreakersDefault.StandardThinking

I do think that if other apps have gotten away with using the literal hardcoded defaults and it seems to work fine.

I think that if someone is looking to use the settings and has no clue about seq_breakers, they are probably wanting those defaults. Like even the thinking model I was looking at, appears to use those 4, despite using the think tag, as they output a newline, it seems they purposefully design models meant to use DRY to work with those 4 defaults, after all DRY is mostly for creative writting.