Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Quantization improvements for k_quants #2707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Aug 22, 2023
Merged

Quantization improvements for k_quants #2707

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f26f9ef
Improve LLaMA-2 2-, 3- and 4-bit quantization
Kawrakow Aug 13, 2023
77aea72
Minor 4-bit quantization improvement
Kawrakow Aug 13, 2023
ec9cb75
Some more fine tuning
Kawrakow Aug 13, 2023
4f8dcb1
Adding make_qkx2_quants
Kawrakow Aug 14, 2023
e9f1340
Another minor improvement
Kawrakow Aug 14, 2023
1c1f985
Q2_K improvement
Kawrakow Aug 14, 2023
404e43c
Iterating
Kawrakow Aug 16, 2023
9f78d4c
Revert Q5_K back to make_qkx1_quants
Kawrakow Aug 20, 2023
e2af308
Better Q6_K
Kawrakow Aug 21, 2023
b706339
make_qkx2_quants is better for Q5_K after all
Kawrakow Aug 22, 2023
35a0b97
Fix after rebasing on master
Kawrakow Aug 22, 2023
fdf73db
Fix for changed tensor names
Kawrakow Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Some more fine tuning
  • Loading branch information
@Kawrakow
Kawrakow committed Aug 22, 2023
commit ec9cb753a66b1eabcbef78f21d82a4785f6f6c4a
2 changes: 1 addition & 1 deletion 2 k_quants.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -811,7 +811,7 @@ void quantize_row_q5_K_reference(const float * restrict x, block_q5_K * restrict
float max_scale = 0; // as we are deducting the min, scales are always positive
float max_min = 0;
for (int j = 0; j < QK_K/32; ++j) {
scales[j] = make_qkx1_quants(32, 31, x + 32*j, L + 32*j, &mins[j], 5, 0.f);
scales[j] = make_qkx1_quants(32, 31, x + 32*j, L + 32*j, &mins[j], 9, 0.5f);
float scale = scales[j];
if (scale > max_scale) {
max_scale = scale;
Expand Down
4 changes: 2 additions & 2 deletions 4 llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3726,7 +3726,7 @@ static void llama_model_quantize_internal(const std::string & fname_inp, const s
else if ((ftype == LLAMA_FTYPE_MOSTLY_Q4_K_M || ftype == LLAMA_FTYPE_MOSTLY_Q5_K_M) &&
use_more_bits(i_attention_wv, n_attention_wv)) new_type = GGML_TYPE_Q6_K;
else if (ftype == LLAMA_FTYPE_MOSTLY_Q4_K_S && i_attention_wv < 4) new_type = GGML_TYPE_Q5_K;
else if (ftype == LLAMA_FTYPE_MOSTLY_Q3_K_S && i_attention_wv < 2) new_type = GGML_TYPE_Q5_K;
else if (ftype == LLAMA_FTYPE_MOSTLY_Q3_K_S && i_attention_wv < 4) new_type = GGML_TYPE_Q5_K;
else if (QK_K == 64 && (ftype == LLAMA_FTYPE_MOSTLY_Q4_K_S || ftype == LLAMA_FTYPE_MOSTLY_Q3_K_S) &&
(i_attention_wv < n_attention_wv/8 || i_attention_wv >= 7*n_attention_wv/8)) new_type = GGML_TYPE_Q6_K;
++i_attention_wv;
Expand All @@ -3740,7 +3740,7 @@ static void llama_model_quantize_internal(const std::string & fname_inp, const s
else if ((ftype == LLAMA_FTYPE_MOSTLY_Q4_K_M || ftype == LLAMA_FTYPE_MOSTLY_Q5_K_M) &&
use_more_bits(i_feed_forward_w2, n_feed_forward_w2)) new_type = GGML_TYPE_Q6_K;
else if (ftype == LLAMA_FTYPE_MOSTLY_Q4_K_S && i_feed_forward_w2 < 4) new_type = GGML_TYPE_Q5_K;
else if (ftype == LLAMA_FTYPE_MOSTLY_Q3_K_S && i_feed_forward_w2 < 2) new_type = GGML_TYPE_Q5_K;
else if (ftype == LLAMA_FTYPE_MOSTLY_Q3_K_S && i_feed_forward_w2 < 4) new_type = GGML_TYPE_Q5_K;
++i_feed_forward_w2;
} else if (name.find("attn_output.weight") != std::string::npos) {
if (ftype == LLAMA_FTYPE_MOSTLY_Q3_K_M || ftype == LLAMA_FTYPE_MOSTLY_Q2_K) new_type = GGML_TYPE_Q4_K;
Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.