[BUG]: System.AccessViolationException on KernelMemory Example in Llama.Examples

[JLeaman99]

Description

When attempting to run the Document Q&A example of the KernelMemory module in Llama.Examples I get a System Access Violation after both sample documents are ingested on v0.20 of LlamaSharp.

I got the same error when using both Llama 3.1 8b instruct & the provided Llama 3.2 1B instruct from the Unit Test

System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:

at LLama.Native.SafeLLamaSamplerChainHandle.g__llama_sampler_sample|4_0(LLama.Native.SafeLLamaSamplerChainHandle, LLama.Native.SafeLLamaContextHandle, Int32)

at LLama.Native.SafeLLamaSamplerChainHandle.Sample(LLama.Native.SafeLLamaContextHandle, Int32)
at LLama.Sampling.BaseSamplingPipeline.Sample(LLama.Native.SafeLLamaContextHandle, Int32)
at LLama.StatelessExecutor+d__24.MoveNext()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext(System.Threading.Thread)
at System.Runtime.CompilerServices.TaskAwaiter+<>c.b__12_0(System.Action, System.Threading.Tasks.Task)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean)
at System.Threading.Tasks.Task.RunContinuations(System.Object)
at System.Threading.Tasks.Task.FinishSlow(Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

<title>Document</title>

	Name	Value	Type
▶	context	{2436473601360}	LLama.Native.SafeLLamaContextHandle
	index	343	int
▶	this	{2436396120624}	LLama.Native.SafeLLamaSamplerChainHandle

Reproduction Steps

1. Clone v0.20 of Repo
2. Run Llama.Examples
3. Select Document Q&A Option
4. Use Debug Llama-3.2-1B-Instruct-Q4_0.gguf as model

Environment & Configuration

• Operating system: Win 10
• .NET runtime version: 8.0
• LLamaSharp version: 0.20
• CUDA version (if you are using cuda backend): N/A running CPU only for now
• CPU & GPU device: Ryzen 7 5700x & RTX 3070

Known Workarounds

No response

[zsogitbe]

I don't see Llama 3.2 support yet... try Llama 3.1 and below.

[martindevans]

Do you know what version of llama.cpp is required for Llama 3.2? We're about 4 weeks out of date right now.

[zsogitbe]

Difficult to see, but I think 794fe23 has it already (about 3-4 weeks ago). It would be great to sync with the last cpp version though, if possible (see other messages).

[JLeaman99]

I don't see Llama 3.2 support yet... try Llama 3.1 and below.

Gotcha gotcha, I did try with Llama 3.1 and I got the same error. I was able to get the Example working on v0.19 but on v0.20 it seems to break.

[martindevans]

Has this been resolved now?

[JLeaman99]

I am still currently running v0.19 on my solution as when I updated to v0.22 / 0.23 I experienced the GGML_ASSERT error seen in Issue #1151.

I was able to get around this System.AccessViolationException by tweaking the model params to have Embeddings = false, UBatchSize = BatchSize and using Llama 3.1

Will close to reduce backlog