How to use the gguf-split / Model sharding demo
#6404
Replies: 4 comments · 20 replies
-
|
Maybe I am wrong, but I couldn't make |
Beta Was this translation helpful? Give feedback.
-
|
@ngxson Here it is. The really strange part is that it looks like changing the parameter very slightly results in drastically different outputs: |
Beta Was this translation helpful? Give feedback.
-
|
Ah, and I'm not so lucky as @maziyarpanahi with splitting Mixtral 8x22b. Only possible difference is that I quantized each model with an imatrix, but I'm not getting nicely sized splits sans flags. |
Beta Was this translation helpful? Give feedback.
-
|
please for the love of gpt how do you merge gguf files? there isn't much docs on the net? I tried compiling llama.cpp on windows and i've got that but now what? Just want to merge 1-5 sharded gguf of command r+! |
Beta Was this translation helpful? Give feedback.
-
|
@4cecoder |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
On Windows you can compile llama.cpp by opening a VS native tools command prompt (i.e. |
Beta Was this translation helpful? Give feedback.
-
|
@dranger003 @phymbert may I ask how to compile gguf-split on MAC? (llamacpp) taozhiyu@603e5f4a42f1 llama.cpp-master % gguf-split |
Beta Was this translation helpful? Give feedback.
-
|
Hum, I am Mac less. @ggerganov, could you please help ? Is not simply calling |
Beta Was this translation helpful? Give feedback.
-
make
./gguf-split |
Beta Was this translation helpful? Give feedback.
-
|
Summary updated ^) |
Beta Was this translation helpful? Give feedback.
-
|
Works on Mac.thanks. 'make |
Beta Was this translation helpful? Give feedback.
-
|
I dont have to compile this whole project just to merge gguf's do i? |
Beta Was this translation helpful? Give feedback.
-
|
Please also include more clear and specific instructions for --merge? |
Beta Was this translation helpful? Give feedback.
-
|
Merging is not necessary anymore in most use cases. |
Beta Was this translation helpful? Give feedback.
-
|
Merging in LLM studio is no longer necessary so just use LLM studio on windows @bennmann |
Beta Was this translation helpful? Give feedback.
-
|
@phymbert "most cases" is not all cases, and it would be nice if this was clearly documented with full example for those cases that fall outside of "most". Front ends that use llama.cpp as a backend are sometimes not coded to operate with split files. and so people using those need to merge themselves. I dont know if it still is, but oobabooga is an example (at least a few weeks ago when I last tested it) where a UI was unable to deal with split files. People are still running into issues trying to use this tool. As an example here's two in the past week. https://huggingface.co/leafspark/Meta-Llama-3.1-405B-Instruct-GGUF/discussions/2 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Context
Distributing and storing GGUFs is difficult for 70b+ models, especially on f16. Lot of issue can happen during file transfers, examples:
Typically, GGUFs need to be transferred from Hugging Face to an internal storage like s3, minio, git lfs, nexus or artifactory, then downloaded by the inference server and stored locally (or on a k8s PvC for example).
Storage solutions and filesystems poorly support large GGUF, typically HF does not support files larger than 50GB.
Such limits also exist on Artifactory.
Solution
We recently introduced
gguf-splitCLI and support the load of sharded GGUFs model inllama.cpp:Download a model
Convert to GGUF F16
python -u convert-hf-to-gguf.py \ ~/.cache/huggingface/hub/models--keyfan--grok-1-hf/snapshots/64e7373053c1bc7994ce427827b78ec11c181b3e/ \ --outfile grok-1-f16.gguf \ --outtype f16NOTE: Follow llama.cpp build instructions to generate all tools/cli:
make.Quantize (optional)
Build model shards
It is possible to use different sharding strategy:
--split-max-tensors 256--split-max-size 48GIt will produce 9 files with maximum 256 tensors in each.
You can then upload the sharded model to your HF Repo:
Files produced by
gguf-splitare valid GGUFs, so you can visualize them in HF.Load sharded model
llama_load_model_from_filewill detect the number of files and will load additional tensors from the rest of files.You may notice:
Load sharded model from a remote URL
Beta Was this translation helpful? Give feedback.
All reactions