Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

server : add speculative decoding support #10455

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 25, 2024
Merged

server : add speculative decoding support #10455

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
156aa6d
server : add speculative decoding support
ggerganov Nov 22, 2024
0ba40c3
server : add helper function slot.can_speculate()
ggerganov Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
server : add helper function slot.can_speculate() 
ggml-ci
  • Loading branch information
@ggerganov
ggerganov committed Nov 25, 2024
commit 0ba40c36150e29d1b7893a35e41f806e43b596e3
9 changes: 6 additions & 3 deletions 9 examples/server/server.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,10 @@ struct server_slot {
return state != SLOT_STATE_IDLE;
}

bool can_speculate() const {
return ctx_dft && params.speculative.n_max > 0 && params.cache_prompt;
}

void add_token(const completion_token_output & token) {
if (!is_processing()) {
SLT_WRN(*this, "%s", "slot is not processing\n");
Expand Down Expand Up @@ -1270,7 +1274,7 @@ struct server_context {
{"min_keep", slot.params.sampling.min_keep},
{"grammar", slot.params.sampling.grammar},
{"samplers", samplers},
{"speculative", slot.params.speculative.model.empty() ? false : true},
{"speculative", slot.can_speculate()},
{"speculative.n_max", slot.params.speculative.n_max},
{"speculative.n_min", slot.params.speculative.n_min},
{"speculative.p_min", slot.params.speculative.p_min},
Expand Down Expand Up @@ -2302,11 +2306,10 @@ struct server_context {
}

// check if the slot supports speculative decoding
if (!slot.ctx_dft || slot.params.speculative.n_max <= 0 || !slot.params.cache_prompt) {
if (!slot.can_speculate()) {
continue;
}

// TODO: configurable through requests
struct common_speculative_params params_spec;
params_spec.n_draft = slot.params.speculative.n_max;
params_spec.n_reuse = llama_n_ctx(slot.ctx_dft) - slot.params.speculative.n_max;
Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.