Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit f94faab

Browse filesBrowse files
committed
Merge branch 'main' of https://github.com/abetlen/llama-cpp-python into main
2 parents 702306b + bce6dc0 commit f94faab
Copy full SHA for f94faab

File tree

Expand file treeCollapse file tree

2 files changed

+6
-4
lines changed
Filter options
Expand file treeCollapse file tree

2 files changed

+6
-4
lines changed

‎docs/server.md

Copy file name to clipboardExpand all lines: docs/server.md
+5-3Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,12 +76,14 @@ Function calling is completely compatible with the OpenAI function calling API a
7676

7777
You'll first need to download one of the available function calling models in GGUF format:
7878

79-
- [functionary-7b-v1](https://huggingface.co/abetlen/functionary-7b-v1-GGUF)
79+
- [functionary](https://huggingface.co/meetkai)
8080

81-
Then when you run the server you'll need to also specify the `functionary` chat_format
81+
Then when you run the server you'll need to also specify either `functionary-v1` or `functionary-v2` chat_format.
82+
83+
Note that since functionary requires a HF Tokenizer due to discrepancies between llama.cpp and HuggingFace's tokenizers as mentioned [here](https://github.com/abetlen/llama-cpp-python/blob/main?tab=readme-ov-file#function-calling), you will need to pass in the path to the tokenizer too. The tokenizer files are already included in the respective HF repositories hosting the gguf files.
8284

8385
```bash
84-
python3 -m llama_cpp.server --model <model_path> --chat_format functionary
86+
python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>
8587
```
8688

8789
Check out this [example notebook](https://github.com/abetlen/llama-cpp-python/blob/main/examples/notebooks/Functions.ipynb) for a walkthrough of some interesting use cases for function calling.

‎examples/notebooks/Functions.ipynb

Copy file name to clipboardExpand all lines: examples/notebooks/Functions.ipynb
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
"The OpenAI compatbile web server in `llama-cpp-python` supports function calling.\n",
1010
"\n",
1111
"Function calling allows API clients to specify a schema that gives the model a format it should respond in.\n",
12-
"Function calling in `llama-cpp-python` works by combining models pretrained for function calling such as [`functionary`](https://huggingface.co/abetlen/functionary-7b-v1-GGUF) with constrained sampling to produce a response that is compatible with the schema.\n",
12+
"Function calling in `llama-cpp-python` works by combining models pretrained for function calling such as [`functionary`](https://huggingface.co/meetkai) with constrained sampling to produce a response that is compatible with the schema.\n",
1313
"\n",
1414
"Note however that this improves but does not guarantee that the response will be compatible with the schema.\n",
1515
"\n",

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.