Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings
Discussion options

devashishraj
Feb 15, 2025

trying to convert embedding into tensor leads to Segmentation fault:

code

import logging
import torch
from llama_cpp import Llama
from rich.console import Console

# Configure logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
console = Console(width=120)

embpath = "all-MiniLM-L6-v2-ggml-model-f16.gguf"
embedModel = Llama(model_path=embpath,embedding=True,verbose=True)

# test embedding model
query = ["Test sentence"]
try:
    embeds = embedModel.embed(input=query)
    print(embeds)
    genAns_tensor = torch.tensor(embeds)
    
    del embedModel
except Exception as e:
    print("Embedding error:", e)
  • code works for only creating embeddings

logs

llama_kv_cache_init: kv_size = 512, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 6, can_shift = 1
llama_kv_cache_init: layer 0: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 1: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 2: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 3: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 4: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 5: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init:      Metal KV buffer size =     4.50 MiB
llama_init_from_model: KV self size  =    4.50 MiB, K (f16):    2.25 MiB, V (f16):    2.25 MiB
llama_init_from_model:        CPU  output buffer size =     0.00 MiB
llama_init_from_model:      Metal compute buffer size =    17.00 MiB
llama_init_from_model:        CPU compute buffer size =     3.50 MiB
llama_init_from_model: graph nodes  = 221
llama_init_from_model: graph splits = 2
Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | MATMUL_INT8 = 1 | ACCELERATE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
Model metadata: {'tokenizer.ggml.cls_token_id': '101', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.seperator_token_id': '102', 'tokenizer.ggml.unknown_token_id': '100', 'tokenizer.ggml.token_type_count': '2', 'general.file_type': '1', 'tokenizer.ggml.eos_token_id': '102', 'bert.context_length': '512', 'bert.pooling_type': '1', 'tokenizer.ggml.bos_token_id': '101', 'bert.attention.head_count': '12', 'bert.feed_forward_length': '1536', 'tokenizer.ggml.mask_token_id': '103', 'tokenizer.ggml.model': 'bert', 'bert.attention.causal': 'false', 'general.name': 'all-MiniLM-L6-v2', 'bert.block_count': '6', 'bert.attention.layer_norm_epsilon': '0.000000', 'bert.embedding_length': '384', 'general.architecture': 'bert'}
Using fallback chat format: llama-2
Fatal Python error: Segmentation fault

Thread 0x0000000204908840 (most recent call first):
  File "/Users/devashishraj/Desktop/localRAG/lrag/lib/python3.13/site-packages/llama_cpp/_internals.py", line 306 in decode
  File "/Users/devashishraj/Desktop/localRAG/lrag/lib/python3.13/site-packages/llama_cpp/llama.py", line [1]    63839 segmentation fault  PYTHONFAULTHANDLER=1 python3 -X dev embeddingTest.py

venv package list

pip list
Package                  Version
------------------------ -----------
aiohappyeyeballs         2.4.4
aiohttp                  3.11.10
aiosignal                1.3.2
annotated-types          0.7.0
anyio                    4.7.0
attrs                    24.3.0
beautifulsoup4           4.12.3
certifi                  2024.12.14
charset-normalizer       3.4.0
dataclasses-json         0.6.7
diskcache                5.6.3
faiss-cpu                1.9.0.post1
filelock                 3.17.0
frozenlist               1.5.0
fsspec                   2025.2.0
gpt4all                  2.8.2
h11                      0.14.0
httpcore                 1.0.7
httpx                    0.28.1
httpx-sse                0.4.0
huggingface-hub          0.28.1
idna                     3.10
Jinja2                   3.1.5
joblib                   1.4.2
jsonpatch                1.33
jsonpointer              3.0.0
langchain                0.3.12
langchain-community      0.3.12
langchain-core           0.3.33
langchain-ollama         0.2.3
langchain-text-splitters 0.3.3
langsmith                0.2.3
llama_cpp_python         0.3.7
markdown-it-py           3.0.0
MarkupSafe               3.0.2
marshmallow              3.23.1
mdurl                    0.1.2
mpmath                   1.3.0
multidict                6.1.0
mypy-extensions          1.0.0
networkx                 3.4.2
numpy                    2.2.3
ollama                   0.4.7
orjson                   3.10.12
packaging                24.2
pillow                   11.1.0
pip                      25.0.1
propcache                0.2.1
pydantic                 2.10.3
pydantic_core            2.27.1
pydantic-settings        2.7.0
Pygments                 2.18.0
PyMuPDF                  1.25.1
python-dotenv            1.0.1
PyYAML                   6.0.2
regex                    2024.11.6
requests                 2.32.3
requests-toolbelt        1.0.0
rich                     13.9.4
safetensors              0.5.2
scikit-learn             1.6.1
scipy                    1.15.1
sentence-transformers    3.4.1
setuptools               75.8.0
sniffio                  1.3.1
soupsieve                2.6
SQLAlchemy               2.0.36
sympy                    1.13.1
tenacity                 9.0.0
threadpoolctl            3.5.0
tiktoken                 0.8.0
tokenizers               0.21.0
torch                    2.6.0
tqdm                     4.67.1
transformers             4.48.3
typing_extensions        4.12.2
typing-inspect           0.9.0
urllib3                  2.2.3
yarl                     1.18.3
You must be logged in to vote

Replies: 1 comment

Comment options

onestardao
Jul 29, 2025

Hey! Looks like you're hitting a classic case of runtime collapse during tensor conversion — often caused by an incompatible backend, unsupported gguf variation, or embedding mode not being fully set up in the wrapper layer.

If helpful, I’ve cataloged this kind of issue as BBCR in the WFGY Problem Map — which covers common LLM-related crashes, including segmentation faults like this one. You can check if your stack matches any known patterns (e.g., embedding=True without full alignment to the gguf tokenizer metadata, or unsafe torch conversion when the embedding layer didn’t return the expected shape/type).

In case you're curious, I also open-sourced the reasoning engine behind this map (MIT license), and the PDF version of it includes a breakdown of the WFGY formula and system-level stabilizers.

Let me know if you'd like a deeper diagnosis — happy to compare stack traces.
You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
@devashishraj @onestardao
Morty Proxy This is a proxified and sanitized view of the page, visit original site.