Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings
Discussion options

gael-vanderlee
May 22, 2023

Hello,
I'm pretty new to all this, apologies if the answer is obvious.
I'm trying to make this (and similar) libraries work locally but they all as the user to load the model weights. I found a model on huggingFace but when loading I get the following error:

>>> LlamaCpp(model_path="models\\ggml-model-q4_0.bin")
  Could not load Llama model from path: [...]models\ggml-model-q4_0.bin. Received error  (type=value_error)

I assume this is is because I downloaded the wrong model. Where do I find them and how do I know which are compatible with this library ?

Thanks for the help !
You must be logged in to vote
Answered by gjmulder May 22, 2023

As of now, the llama models and their derivatives are licensed for restricted distribution by Facebook, so they will never be distributed from or linked to in this repo.

Please refer to the llama.cpp README.md for more information. Or you could try Googling site:huggingface.co+llama+ggml...

Note that there are currently three versions of models compatible with different versions of python-llama-cpp. You would want to install the latest python-llama-cpp version and use the latest (ggml v3) llama.cpp models to ensure both the best compatibility and performance.

HuggingFace is now providing a leaderboard of the best quality models. Note again, however that the models linked off the leaderboa…

View full answer

Replies: 4 comments · 6 replies

Comment options

gjmulder
May 22, 2023

As of now, the llama models and their derivatives are licensed for restricted distribution by Facebook, so they will never be distributed from or linked to in this repo.

Please refer to the llama.cpp README.md for more information. Or you could try Googling site:huggingface.co+llama+ggml...

Note that there are currently three versions of models compatible with different versions of python-llama-cpp. You would want to install the latest python-llama-cpp version and use the latest (ggml v3) llama.cpp models to ensure both the best compatibility and performance.

HuggingFace is now providing a leaderboard of the best quality models. Note again, however that the models linked off the leaderboard are not directly compatible with llama.cpp (and therefore python-llama-cpp). You can, again with a bit of searching, find the converted ggml v3 llama.cpp equivalent models.
You must be logged in to vote
0 replies
Answer selected by gael-vanderlee
Comment options

gael-vanderlee
May 22, 2023
Author

Got it, thanks for the response!
You must be logged in to vote
0 replies
Comment options

gael-vanderlee
May 22, 2023
Author

It seems that I'm still getting the following error:

llama.cpp: loading model from \models\llama-13b-ggml-v3-q4_0.bin
error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?
llama_init_from_file: failed to load model
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2022.2.3\plugins\python\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
__root__
  Could not load Llama model from path: \models\llama-13b-ggml-v3-q4_0.bin. Received error  (type=value_error)

Is this an error with the model file or does the issue lie somewhere else ?
You must be logged in to vote
5 replies
@gjmulder
Comment options

gjmulder May 22, 2023

pip list | grep llama ?
@gael-vanderlee
Comment options

gael-vanderlee May 22, 2023
Author 

Package                 Version
----------------------- -----------
aiohttp                 3.8.4
charset-normalizer      3.1.0
chromadb                0.3.23
click                   8.1.3
clickhouse-connect      0.5.24
colorama                0.4.6
colorclass              2.2.2
commonmark              0.9.1
compressed-rtf          1.0.6
cryptography            40.0.2
dataclasses-json        0.5.7
Deprecated              1.2.13
duckdb                  0.8.0
easygui                 0.98.3
ebcdic                  1.1.1
et-xmlfile              1.1.0
extract-msg             0.41.1
fastapi                 0.95.2
filelock                3.12.0
frozenlist              1.3.3
fsspec                  2023.5.0
greenlet                2.0.2
h11                     0.14.0
hnswlib                 0.7.0
httpcore                0.16.3
httptools               0.5.0 
httpx                   0.23.3
huggingface-hub         0.14.1
idna                    3.4
IMAPClient              2.3.1
Jinja2                  3.1.2
joblib                  1.2.0
langchain               0.0.171
lark-parser             0.12.0
llama-cpp-python        0.1.50
lxml                    4.9.2
lz4                     4.3.2
Markdown                3.4.3
MarkupSafe              2.1.2
marshmallow             3.19.0
marshmallow-enum        1.5.1
monotonic               1.6
mpmath                  1.3.0
msg-parser              1.2.0
msoffcrypto-tool        5.0.1
multidict               6.0.4
mypy-extensions         1.0.0
networkx                3.1
nltk                    3.8.1
numexpr                 2.8.4
numpy                   1.23.5
olefile                 0.46
oletools                0.60.1
openapi-schema-pydantic 1.2.4
openpyxl                3.1.2
packaging               23.1
pandas                  1.5.3
pandoc                  2.3
pcodedmp                1.2.6
pdfminer.six            20221105
Pillow                  9.5.0
pip                     23.1.2
plumbum                 1.8.1
ply                     3.11
posthog                 3.0.1
pycparser               2.21
pydantic                1.10.7
Pygments                2.15.1
pygpt4all               1.1.0
pygptj                  2.0.3
pyllamacpp              2.3.0
pypandoc                1.11
pyparsing               2.4.7
python-dateutil         2.8.2
python-docx             0.8.11
python-dotenv           1.0.0
python-magic            0.4.27
python-pptx             0.6.21
pytz                    2023.3
pytz-deprecation-shim   0.1.0.post0
pywin32                 306
PyYAML                  6.0
red-black-tree-mod      1.20
regex                   2023.5.5
requests                2.30.0
rfc3986                 1.5.0
rich                    13.0.1
RTFDE                   0.0.2
scikit-learn            1.2.2
scipy                   1.10.1
sentence-transformers   2.2.2
sentencepiece           0.1.99
setuptools              65.5.1
six                     1.16.0
sniffio                 1.3.0
soupsieve               2.4.1
SQLAlchemy              2.0.15
starlette               0.27.0
sympy                   1.12
tabulate                0.9.0
tenacity                8.2.2
threadpoolctl           3.1.0
tokenizers              0.13.3
torch                   2.0.1
torchvision             0.15.2
tqdm                    4.65.0
transformers            4.29.2
typer                   0.9.0
typing_extensions       4.5.0
typing-inspect          0.8.0
tzdata                  2023.3
tzlocal                 4.2
unstructured            0.6.6
urllib3                 2.0.2
uvicorn                 0.22.0
watchfiles              0.19.0
websockets              11.0.3
wheel                   0.38.4
win-unicode-console     0.5
wrapt                   1.14.1
XlsxWriter              3.1.1
yarl                    1.9.2
zstandard               0.21.0

llama-13b-ggml-v3-q4_0.bin
@gjmulder
Comment options

gjmulder May 22, 2023

You forgot the grep:

llama-cpp-python 0.1.50

This version (from memory) supports v2.

Try to upgrade to the latest 0.1.53, which supports v3:

pip install --force-reinstall --ignore-installed --no-cache-dir llama-cpp-python
@gael-vanderlee
Comment options

gael-vanderlee May 22, 2023
Author

Sorry about the grep, I'm on windows on this machine.
Your fix worked, tysm!
@frankandrobot
Comment options

frankandrobot Jun 29, 2023

i'm getting the same error as the OP but am on llama-cpp-python 0.1.63. Trying to load https://huggingface.co/TheBloke/LLaMa-13B-GGML. Every file there gives this or a simlar error

It looks like it was a git lfs issue. However, when I tried llama-13b.ggmlv3.q6_K.bin which did download properly, getting this error:

ggml_metal_add_buffer: buffer 'data' size 10678861824 is larger than buffer maximum of 8589934592
llama_init_from_file: failed to add buffer

Does this only support 4-bit models?
Comment options

Kan-wal
Jun 16, 2023

from where i can found the v3 ggml model?
You must be logged in to vote
1 reply
@frankandrobot
Comment options

frankandrobot Jun 29, 2023

https://huggingface.co/TheBloke/LLaMa-13B-GGML

but see above... getting an error for random GGML v3 files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
documentation Improvements or additions to documentation good first issue Good for newcomers
4 participants
@gael-vanderlee @frankandrobot @gjmulder @Kan-wal
Morty Proxy This is a proxified and sanitized view of the page, visit original site.