[BUG]: KernelMemory.AskAsync() does not work - exception: object reference not set to an instance of an object

[aropb]

Description

I use KernelMemory. LogiBits is empty.

The error occurs at the time of the call:
memory.AskAsync()

Debug with clone classes: BaseSamplingPipeline, DefaultSamplingPipeline

Reproduction Steps

The error occurs at the time of the call:

MemoryAnswer answer = await memory.AskAsync(question: question, filters: filters);

Environment & Configuration

• Operating system: Windows 10/11
• .NET runtime version: 8.0.7
• LLamaSharp version: 0.15.0
• KernelMemory: 0.70.240803.1, 0.69.240727.1
• CUDA version (if you are using cuda backend): -
• CPU & GPU device: CPU Intel Core Ultra 9

Known Workarounds

[jwangga]

I encountered the same issue when running the sample code: "Kernel Memory: Document Q&A" or "Kernel Memory: Save and Load" from the LLama.Examples project.

[tusharmevl]

@aropb @jwangga I am facing the same issue running the example 'Kernel Memory: Document Q&A'. Did you find a fix for this? I am trying to implement a RAG system using this. Is there any other way to implement it apart from Kernel Memory?

[jwangga]

@tusharmevl I have not found a fix for the Kernel Memory issue. It seems that the integration with Semantic Kernel Memory works. You may try using that as an alternative if your system only needs to support Text.

[tusharmevl]

@jwangga Ok Thanks! Yes I need to support text only for now, will try that.

[GalactixGod]

Thanks @jwangga !!

I'm seeing that you can use Semantic Kernel Memory (SKM) as well.

Doesn't appear that you can "chat" with SKM to discuss results unfortunately. Have you been able to figure out a way to "ask" questions of SKM?

[nicholusi2021]

I'm also having the same issue with the Kernel Memory: Document Q&A example.

[aropb]

Please, I really need to fix the error.

So far, I can only use such versions:
Microsoft.KernelMemory.Core = 0.62.240605.1
LLamaSharp = 0.13.0

Any newer versions do not work.

I think the mistake is here:

ISamplingPipelineExtensions.Sample()
...
var span = CollectionsMarshal.AsSpan(lastTokens);
--->!
return pipeline.Sample(ctx, logits, span);
...

[aropb]

I found the place where the error occurs.

llama_get_logits_ith suddenly return null.
// returns NULL for invalid ids.

    public Span<float> SafeLLamaContextHandle.GetLogitsIth(int i)
    {
        var model = ThrowIfDisposed();

        unsafe
        {
            var logits = llama_get_logits_ith(this, i);
            return new Span<float>(logits, model.VocabCount);
        }
    }

Stack:

StatelessExecutor.InferAsync()
...
var id = pipeline.Sample(Context.NativeHandle, Context.NativeHandle.GetLogitsIth(_batch.TokenCount - 1), lastTokens);
...

But I don't understand what to do next and how to fix the error. Apparently, null shouldn't be there. Can anyone help with this error?
Due to this error, it is impossible to use Kernel memory.

Thanks.

[martindevans]

That's probably indicative of two bugs in LLamaSharp.

Wrapper Error

The docs for llama_get_logits_ith (see here) say:

// Logits for the ith token. For positive indices, Equivalent to:
// llama_get_logits(ctx) + ctx->output_ids[i]*n_vocab
// Negative indicies can be used to access logits in reverse order, -1 is the last logit.
// returns NULL for invalid ids.
LLAMA_API float * llama_get_logits_ith(struct llama_context * ctx, int32_t i);

So it is valid for llama_get_logits_ith to return null! That means this SafeLLamaContextHandle.GetLogitsIth is incorrectly written, it should check for null and raise some kind of error in that case (throw an exception most likely). It is never valid to pass a null pointer into a span constructor!

This is why you get a hard crash instead of an exception.

Higher Level Error

llama_get_logits_ith returns null if an invalid value for i is passed in. There must be a bug somewhere in the higher level that is causing an incorrect value to be passed in. Since this error only seems to affect kernel memory it must be something specific to the KM wrapper.

[aropb]

@martindevans I have found a solution.

1. Embeddings = false

   ...
   public static IKernelMemoryBuilder WithLLamaSharp(this IKernelMemoryBuilder builder, LLamaSharpConfig config)
    {
        ModelParams parameters = new(config.ModelPath)
        {
            Embeddings = false,
            ...
   

2. set the values: UBatchSize, BatchSize
   ...
   public LLamaSharpTextEmbeddingGenerator(LLamaSharpConfig config, LLamaWeights weights)
   {
   ModelParams @params = new(config.ModelPath)
   {
   Embeddings = true,
   ...
   UBatchSize = 2000,
   BatchSize = 2000
   };