`server`: streaming of tool calls and thoughts when `--jinja` is on#12379

Merged

ochafik merged 102 commits intoggml-org:masterfrom

ochafik:tool-diffs

May 25, 2025

Commits on Mar 12, 2025

add common_regex w/ support for partial final matches
ochafik
committed
16c9c63
Copy full SHA for 16c9c63

Commits on Mar 13, 2025

add common_json w/ support for truncated json healing
ochafik
committed
6dcff43
Copy full SHA for 6dcff43
renaming: string_find_partial_stop (moved to common.cpp)
ochafik
committed
a95fe78
Copy full SHA for a95fe78
add common_chat_msg_diff
ochafik
committed
ce2f593
Copy full SHA for ce2f593
partial common_chat_parse
Olivier Chafik
committed
cd3681d
Copy full SHA for cd3681d
refactor parser w/ optionals
ochafik
committed
9462365
Copy full SHA for 9462365
server: wire chat diffs in stream mode
ochafik
committed
6ed8a8f
Copy full SHA for 6ed8a8f
fix trigger of thinking models (must happen after thoughts are closed)
Olivier Chafik
committed
eaeed7d
Copy full SHA for eaeed7d

Commits on Mar 14, 2025

nits + docs
ochafik
committed
d6e680a
Copy full SHA for d6e680a
fix functionary v3.2 raw python!
ochafik
committed
64ea080
Copy full SHA for 64ea080
rename: common_chat_syntax (now contains format)
ochafik
committed
c46d4da
Copy full SHA for c46d4da
rm common_regex.at_start
ochafik
committed
4358d5d
Copy full SHA for 4358d5d
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
f477288
Copy full SHA for f477288
fix gcc compilation
ochafik
committed
e0202b3
Copy full SHA for e0202b3
fix unreachable code warning after [[noreturn]] annotation
ochafik
committed
f840e3a
Copy full SHA for f840e3a
fix / refactor test-regex-partial
ochafik
committed
af7391e
Copy full SHA for af7391e
fix test-chat
ochafik
committed
449917b
Copy full SHA for 449917b
rm spaces
ochafik
committed
b428b5c
Copy full SHA for b428b5c
fix command r7b partial parsing (lacked args path)
ochafik
committed
668fc90
Copy full SHA for 668fc90
Update test_chat_completion.py
ochafik
committed
b48ab23
Copy full SHA for b48ab23

Commits on Mar 15, 2025

refactor + test chat parser (try_consume_json_with_dumped_args, literal based thinking tags parsing)
ochafik
committed
aefc8a4
Copy full SHA for aefc8a4
return partial msg from server
ochafik
committed
22428a4
Copy full SHA for 22428a4
refactor partial json
ochafik
committed
5b9c5a4
Copy full SHA for 5b9c5a4
don't return empty <think></think>
ochafik
committed
3fbe84f
Copy full SHA for 3fbe84f
test_tool_call: allow comment lines in now-multiline code strings (for functionary v3.2)
ochafik
committed
d4cb7fe
Copy full SHA for d4cb7fe
accommodate yet another deepseek r1 distill fantasy syntax (<｜tool▁calls｜>)
ochafik
committed
31f5eb2
Copy full SHA for 31f5eb2
rm space
ochafik
committed
bddc65a
Copy full SHA for bddc65a
nit: fix python type
ochafik
committed
ea3bf03
Copy full SHA for ea3bf03
refactor test-chat-parser
ochafik
committed
f3bfbc6
Copy full SHA for f3bfbc6
fix QwQ 32B tool call parsing after thoughts (hermes2)
ochafik
committed
bb7b9fe
Copy full SHA for bb7b9fe
fix thinking models + tool calls (</think> not part of trigger's capture!)
ochafik
committed
f0ea330
Copy full SHA for f0ea330
reinstate tool call id logic, keep track of previously generated ids
ochafik
committed
7856949
Copy full SHA for 7856949
better logs for triggers
ochafik
committed
2412b5d
Copy full SHA for 2412b5d
fix msg diff test
ochafik
committed
02913b0
Copy full SHA for 02913b0
try_consume_regex: basic tests + fix non-partial case
ochafik
committed
c5c3482
Copy full SHA for c5c3482
chat-parser: test+fix finish, incomplete methods
ochafik
committed
af79da0
Copy full SHA for af79da0
normalize args in test-chat
ochafik
committed
562800f
Copy full SHA for 562800f
consume spaces after parse_json_tool_calls
ochafik
committed
ddeb318
Copy full SHA for ddeb318
Revert "fix thinking models + tool calls (</think> not part of trigger's capture!)"
Show description for 6c3f87e
ochafik
committed
6c3f87e
Copy full SHA for 6c3f87e
fix required tool calls w/ thinking models that have pre-opened thinking tags
ochafik
committed
e2cef66
Copy full SHA for e2cef66
fix thinking model's initial trigger (take 2) + test qwq's template
ochafik
committed
7a61eca
Copy full SHA for 7a61eca
refactor chat parser (rm incomplete)
ochafik
committed
2f55571
Copy full SHA for 2f55571
test groups of common_chat_msg_parser.try_consume_regex
ochafik
committed
303f640
Copy full SHA for 303f640
run most test_tool_call tests in stream + non-stream modes
ochafik
committed
e9540ad
Copy full SHA for e9540ad

Commits on Mar 16, 2025

make functionary v3.2 parsing more strict (differentiate first match from others)
ochafik
committed
a818114
Copy full SHA for a818114
send final diff from server, to close off raw python arguments
ochafik
committed
5031366
Copy full SHA for 5031366
nit: spaces
ochafik
committed
dae6a28
Copy full SHA for dae6a28
fix diff aggregation logic in make_any_request
ochafik
committed
f026cb0
Copy full SHA for f026cb0
fix test_chat_completion_with_timings_per_token & test_logprobs_stream
ochafik
committed
e7f9d3e
Copy full SHA for e7f9d3e
add missing functional import for gcc compilation
ochafik
committed
165b525
Copy full SHA for 165b525
fix typo in test_calc_result
ochafik
committed
9d4a6f1
Copy full SHA for 9d4a6f1
fix thoughts parsing logic
ochafik
committed
64b4039
Copy full SHA for 64b4039
support partial content streaming in Generic mode
ochafik
committed
fbba5da
Copy full SHA for fbba5da
strip reasoning (now that tags are strings and not regexes)
ochafik
committed
4dcd653
Copy full SHA for 4dcd653
run test_thoughts in stream mode too
ochafik
committed
56156b7
Copy full SHA for 56156b7
r1: avoid partial call triggers from spaces
ochafik
committed
5dfa2f7
Copy full SHA for 5dfa2f7
fix test_thoughts / refactor expectations
ochafik
committed
91a5084
Copy full SHA for 91a5084
fix partial json crashes
ochafik
committed
4f78d44
Copy full SHA for 4f78d44
fix test-chat's unparsed thought expectation
ochafik
committed
ea57e47
Copy full SHA for ea57e47

Commits on Mar 23, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
1d25178
Copy full SHA for 1d25178
fix partial json crash after comma
ochafik
committed
42cb16f
Copy full SHA for 42cb16f
fix test-chat.cpp
ochafik
committed
37b4a3a
Copy full SHA for 37b4a3a
fix gcc build of test
ochafik
committed
13d725d
Copy full SHA for 13d725d

Commits on Mar 26, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
a40aead
Copy full SHA for a40aead

Commits on Apr 1, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
329d943
Copy full SHA for 329d943

Commits on Apr 3, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
e63e542
Copy full SHA for e63e542

Commits on Apr 4, 2025

fix regex-partial (drop reluctant repetitions conversions)
ochafik
committed
21cd34c
Copy full SHA for 21cd34c
partial regex: allow newlines in prefixes
ochafik
committed
5f0450d
Copy full SHA for 5f0450d
tool-call: allow content prelude before hermes2 tool calls (for Qwen2.5)
ochafik
committed
36ecb01
Copy full SHA for 36ecb01
Update function-calling.md
ochafik
committed
68eeff1
Copy full SHA for 68eeff1
nit: spaces
ochafik
committed
12deff6
Copy full SHA for 12deff6
Update tool_bench.py
ochafik
committed
d0a686b
Copy full SHA for d0a686b
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
a604b2d
Copy full SHA for a604b2d

Commits on Apr 5, 2025

Inject date_string in llama 3.x + test it & functionary v2
Show description for 90789cd
ochafik
committed
90789cd
Copy full SHA for 90789cd

Commits on Apr 7, 2025

Inject date_string in llama 3.x + fix for functionary v2
Show description for 71435cf
ochafik
committed
71435cf
Copy full SHA for 71435cf
add missing chrono include
ochafik
committed
543b73e
Copy full SHA for 543b73e
move/fix detection of functionary v3.1 before llama 3.x, fix & test their non-tool mode
ochafik
committed
e3c372c
Copy full SHA for e3c372c
Merge branch 'date' into tool-diffs
ochafik
committed
387611a
Copy full SHA for 387611a
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
01a3e31
Copy full SHA for 01a3e31
move string_find_partial_stop & string_ends_with to common
ochafik
committed
59b87c5
Copy full SHA for 59b87c5
add common_regex (supports partial matches)
ochafik
committed
ff35374
Copy full SHA for ff35374
Update test-regex-partial.cpp
ochafik
committed
869e1a9
Copy full SHA for 869e1a9

Commits on Apr 18, 2025

Update common/common.cpp
Show description for 6f109fa

ochafik
and
ggerganov
authored
6f109fa
Copy full SHA for 6f109fa
Update common/regex-partial.cpp
Show description for 908e12f

ochafik
and
ggerganov
authored
908e12f
Copy full SHA for 908e12f
Update common/regex-partial.cpp
Show description for 868b442

ochafik
and
ggerganov
authored
868b442
Copy full SHA for 868b442
Update common/regex-partial.h
Show description for 2ea5f5c

ochafik
and
ggerganov
authored
2ea5f5c
Copy full SHA for 2ea5f5c
partial regex: add missing iterator end checks
ochafik
committed
b275da3
Copy full SHA for b275da3
string utils: use string_views
ochafik
committed
9b620e5
Copy full SHA for 9b620e5
direct throw to avoid ggml.h include
ochafik
committed
5c99bdc
Copy full SHA for 5c99bdc
regex-partial: replace missed ggml_asserts
ochafik
committed
e051be6
Copy full SHA for e051be6

Commits on May 14, 2025

Merge remote-tracking branch 'origin/master' into partial-regex
ochafik
committed
afce553
Copy full SHA for afce553
Merge branch 'partial-regex' into tool-diffs
ochafik
committed
c879a57
Copy full SHA for c879a57

Commits on May 15, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
ad07a3b
Copy full SHA for ad07a3b
fix merge
ochafik
committed
573e8c3
Copy full SHA for 573e8c3
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
d6e1d5b
Copy full SHA for d6e1d5b
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
6946a83
Copy full SHA for 6946a83

Commits on May 16, 2025

chat-parser: remove input from exception (llm output may contain PII)
ochafik
committed
224101b
Copy full SHA for 224101b
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
6ddda10
Copy full SHA for 6ddda10
disable failing tests from test_tool_call.py
ochafik
committed
8886c24
Copy full SHA for 8886c24

Commits on May 17, 2025

json-partial: add comments
ochafik
committed
810c4c3
Copy full SHA for 810c4c3

Commits on May 23, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
f0d5df2
Copy full SHA for f0d5df2

Commits on May 24, 2025

Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik
committed
40951c8
Copy full SHA for 40951c8