Closed
Description
Using latest master branch on OSX Sonoma 14.2.1 (23C71), M1 Macbook Pro.
When parsing grammars that contain left-recursion, the application parses with a segfault.
Example:
./main -m ./models/llama-2-7b.Q4_0.gguf --grammar "root ::= root \"a\" | \"b\""
Results in:
system_info: n_threads = 8 / 10 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 |
sampling:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 2048, n_predict = -1, n_keep = 1
[1] 50441 segmentation fault ./main -m ./models/llama-2-7b.Q4_0.gguf --grammar
The segfault occurs in llama_grammar_init -- still digging into it, but wanted to at least log the bug.
This is not a regression, and is different from previous grammar-related segfault errors in the past.
This probably affects the server, and represents a vector to crash the server with a malformed request.
Ideally we should be able to catch left-recursion situations and hopefully either continue gracefully, or at the very least fail with an intelligent error message.
ochafik
Metadata
Metadata
Assignees
Labels
Something isn't workingSomething isn't working