Replies: 24 comments 43 replies
-
|
I'm surprised that importance matrices trained on 10k/100k/1m token barely seem to diverge from each other. While some “overfitting” does seem to occur, it's also significantly less prevalent than one might expect. |
Beta Was this translation helpful? Give feedback.
-
|
I think the median / average might not be the smartest measurement. Here's KLD_99 sorted for your q2_K files: Btw, thank you for helping investigate this, I've been very curious about optimal quantization calibration. |
Beta Was this translation helpful? Give feedback.
-
|
Working on a script right now that will automatically quantize a bunch of randomized groups of text data and measure KLD_99. |
Beta Was this translation helpful? Give feedback.
-
|
Please share the script, if you can! |
Beta Was this translation helpful? Give feedback.
-
|
@Artefact2 I have a question. |
Beta Was this translation helpful? Give feedback.
-
|
group_40.txt If I use the first 25k tokens of this data: If I use 40k tokens of Wikitext: I am measuring KL divergence over 30,000 tokens that uses a mix of data (Lyrics, conversations, a wikipedia article or two, etc), so the sample size should be large enough to rule out any differences there. What especially improves are the harder to predict outliers, as noted by KLD_99 and KLD_95. Both are q2_K quantized using the base model of Fett-uccine-7B-GGUF. What doesn't improve over Wikitext is PURELY random data (it's actually a little bit worse); pseudo-random data seems to be optimal, though. |
Beta Was this translation helpful? Give feedback.
-
What's the argument for using an unrelated model instead of the F16 model that you are quantizing? |
Beta Was this translation helpful? Give feedback.
-
No particular one; as long as the data is sufficiently high entropy while still vaguely coherent, this technique should be roughly similar in performance. It might be even better to generate from the source model. |
Beta Was this translation helpful? Give feedback.
-
|
This is about ~50k pseudo-random tokens. I recommend using this file for doing imatrix calibration from here on out; imatrix data should generally transfer well across different models. |
Beta Was this translation helpful? Give feedback.
-
|
@Artefact2 Nice work! On using (pseudo-) random data for imatrix generationIf you have a meaningful calibration dataset, I recommend against (pseudo-)random data. A more comprehensive evaluation that does not rely on a single, quite small dataset will tend to favor imatrix created from textual data. Here is an example:
I'm not using Winogrande and ARC because the test/validation datasets that I have available for these are too small to reveal statistically significant differences. The table summarizes the results
My take from this data
|
Beta Was this translation helpful? Give feedback.
-
|
Out of curiosity, are you running imatrix with the default settings besides |
Beta Was this translation helpful? Give feedback.
-
One thing that I did note is that pretraining style data (non-wikitext) outperformed Wikitext (especially for maximum KL div error reduction), but didn't quite reach the pseudo-random data's KL divergence statistics. Also, if attention tensors are sensitive to calibration data, it might be optimal to use full context for calibration instead of 512 ctx. The main reason why I'm suspicious on wikitext is that it doesn't contain any code or programming examples and I think that skews it towards natural english. |
Beta Was this translation helpful? Give feedback.
-
|
I am using 200k worth of tokens randomly sampled from MiniPile, a decent small pretraining dataset, and I'm comparing different context sizes for calibration (q4_K_S, 7b): 2048 ctx calibration, 200k tokens ish pretraining data 4096 ctx calibration, 200k tokens ish pretraining data 8192 ctx calibration, 200k tokens ish pretraining data 512 ctx calibration, 200k tokens ish pretraining data It would seem that native context size is the best all around for evaluating the importance matrix and results in the lowest average divergence, as well as the best outlier error reduction. |
Beta Was this translation helpful? Give feedback.
-
|
groups_merged.txt Next up, I will be looking into a way to preselect individual pieces of a larger dataset for higher KL div / higher PPL outlier sections so that the quantization is more robust to outliers (instead of throwing random data at it). |
Beta Was this translation helpful? Give feedback.
-
|
Update - we don't need to use "--chunks 1000000" |
Beta Was this translation helpful? Give feedback.
-
|
@InferenceIllusionist Made a new one: |
Beta Was this translation helpful? Give feedback.
-
|
@Dampfinchen off the top looks like enhancedv3 performed better than groups_merged in Mean KLD and Max KLD (Max is much lower). |
Beta Was this translation helpful? Give feedback.
-
|
@InferenceIllusionist thank you again for these results! Seems like V3 does worse than V2 in this test. It would be really interesting to see results for MMLU and wikitext instead of arc-challenge. If it's not too much trouble, could you compare V3, V2, Kalo's original and Turbo-Mini against those? I would really love to know if the differences between these datasets vary against different tests. And this time, perhaps we should look at how 8B Instruct does and not the base model, because Instruct models are what people actually going to use! I know that's asking for a lot of data though, so feel free to take your sweet time with it if you decide to do it! |
Beta Was this translation helpful? Give feedback.
-
anyone tested the v4.1 and v5-rc from https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c ? it is claimed to improve on v3 |
Beta Was this translation helpful? Give feedback.
-
There are indeed chunks that have higher outlier frequencies compared to the rest of the data. This is (500k?) tokens ish of wiki.train.raw. I used "KLD_95" (so, the average KL div for the top 5% highest KL divergences). (I clustered them for chart readability.) As you can see, there are a lot of chunks on the "tail end" that are less affected by quantization: the top 5% highest KL div was 0.937 for the very worst chunk, and 0.214 for the best chunk. My theory is that selecting the top % of chunks that are most "damaged" by quantization for calibration reduces outlier error better than pseudo-random data would. And this way, we have a more mathematically "precise" way of finding better calibration data compared to just throwing 100k tokens of either unsorted Wikitext or gibberish at it. |
Beta Was this translation helpful? Give feedback.
-
|
Pre-selecting in this manner does indeed improve the quantization end result. No imatrix q4_0: ===== KL-divergence statistics Top ~40k tokens, sorted by top5% highest avg kl divergence imatrix, q4_0 ===== KL-divergence statistics All ~500k tokens wikitext data, imatrix, q4_0: ===== KL-divergence statistics |
Beta Was this translation helpful? Give feedback.
-
|
@ikawrakow , @Artefact2 , considering the benefits of the iMatrix, not only in English, but also in other languages like my very own French, could one of you guys assemble and share a training file of alternated English / French (and why not with most of the languages broadly supported to some extend by the Llama2 & Mistral models) sequences of text allowing to train properly an iMatrix benefiting all the languages involved? And an eval file of each language in the fashion of wiki.test.raw? Ideally, it would work like wiki.train.raw does no matter the ctx chosen (I use 32 and it works quite well, but 128 or 512 are still probably a bit better) and the numbers of chunks chosen, up to a few thousands by language. For example, if I set 500 chunks on the iMatrix, 250 chunks would be trained in English, 250 chunks in French. I lack of the know how to do that properly and efficiently involving all the aforementioned criteria. But I think it would help greatly a lot of people to make a single iMatrix file and a single series of quant benefiting a maximum amount of people, including yours truly. |
Beta Was this translation helpful? Give feedback.
-
|
is there data that shows the difference between the 20k random dataset and the new pseudo-random data? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, there is data that supports the current assumption that the pure random dataset leads to worse results than the new balanced random data. However, there is also an indication that we are looking at noisy results and that the new balanced random data is also not optimal. Here is one of the Reddit threads with test results. |
Beta Was this translation helpful? Give feedback.
-
|
Also wondering! From
Also wondering this. At least in theory it could have a big impact what is used. I guess the only way to really test this idea is to create the same model with different data sources for the imatrix and compare ppl? Or is there another way to do this? |
Beta Was this translation helpful? Give feedback.
-
|
In addition to the link in the reply above there are more test results with perplexity and hellaswag in this Reddit thread. Be sure to expand the whole thing, as Reddit likes collapsing replies. |
Beta Was this translation helpful? Give feedback.
-
|
@ikawrakow Thanks for all the hard work, great job. The link to wiki.train.raw is down at https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=blog.salesforceairesearch.com do you know an alternate link, thanks. |
Beta Was this translation helpful? Give feedback.
-
|
I have added them to this repository: https://huggingface.co/datasets/ikawrakow/validation-datasets-for-llama.cpp |
Beta Was this translation helpful? Give feedback.
-
|
Thanks again, what context length would you recommend for creating the imatrix.dat file using wiki-train.raw for 70b models and mixtral? |
Beta Was this translation helpful? Give feedback.
-
|
I use 512 tokens most of the time. There has been a lot of discussion around the imatrix topic, but for me the small improvements that one may get from using different context lengths, different calibration datasets, etc., are very minor compared to what one gains from using vs not using an imatrix. |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone experimented with adding a small value to the importance matrix weights? If I have understood correctly, the importance matrix weights are an approximate diagonal Hessian. If that is the case, a common way to deal with overfitting is to add a scalar multiple of the identity matrix to the (diagonal or otherwise) Hessian (see: Tikhonov_regularization). The regularized Hessian matrix where To determine the optimal value of
From a Bayesian perspective:
It's likely the use of random and semi-random data mentioned in this thread is acting as a "quick and dirty" form of regularisation anyway: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-tikhonov-nc-95.pdf and IMO, it would probably be better to consider doing it in a more principled way - especially considering the calibration dataset is so small and the imatrix computation isn't using the full context nor the correct prompt format, etc. Also, has anyone actually looked into a full Hessian approximation or at least checked the off-diagonals are small? If there is any significant multicollinearity among the weights then this method of using only the diagonal Hessian like this could have a serious impact on the model. How feasible is it to compute the full Hessian approximation by summing |
Beta Was this translation helpful? Give feedback.
-
|
This is mentioning calculating the full hessian on the matrices (which, correct me if I am wrong) over each attention head .. so for example in a 2b model with 8 kv heads that's (very roughly) 8 of 250mx250m matrices (about eight 62.5b 32-bit datasets we need to process in memory). in a 70b model it might be 24 of 342m^2 matrices or ~117b 32-bit datasets. This sort of work is doable in a professional environment with multiple gpus, but not for most people. |
Beta Was this translation helpful? Give feedback.
-
lol |
Beta Was this translation helpful? Give feedback.
-
People put a lot of weight on it being English (and not code or another language) but I think the biggest problem is the data leakage between If you don't know what I mean by "data leakage" then this might help: https://gwern.net/tank There are ways we could correct for this bias in the reported drops in PPL, but most are nonparametric and would require quite a lot of extra computation... |
Beta Was this translation helpful? Give feedback.
-
I was just laughing at wikipedia being "factual." I know what was meant. It was a half joke. On a serious note though, we probably shouldn't be using wikipedia (I'm currently using it too). I don't know how much the gross political bias matters for imatrices, but these LLM's being trained on things like wikipedia and reddit concerns me |
Beta Was this translation helpful? Give feedback.
-
|
Just trying to wrap my head around how these weights are used: So So if and the unweighted version is just: So I'm trying to see how What is special about We can also look at |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I don't get it either. I hope there's documentation and some math formulas to explain it. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, sadly I never did get any further with figuring this out - I did try a few other things that seemed to related better to |
Beta Was this translation helpful? Give feedback.
-
|
Edit: Never mind. Not using the latest llama.cpp with the latest quantizations of Llama 3 8b Instruct and with the right settings were the cause the hallucination spike I was seeing. |
Beta Was this translation helpful? Give feedback.
-
Nope, I've looked at this all sorts of ways and can't for the life of me see where the use of static void quantize_row_q4_0_impl(const float * restrict x, block_q4_0 * restrict y, int64_t n_per_row, const float * quant_weights) {
static_assert(QK4_0 == 32, "QK4_0 must be 32");
if (!quant_weights) {
quantize_row_q4_0_reference(x, y, n_per_row);
return;
}
float weight[QK4_0];
int8_t L[QK4_0];
float sum_x2 = 0;
for (int j = 0; j < n_per_row; ++j) sum_x2 += x[j]*x[j];
float sigma2 = sum_x2/n_per_row;
const int64_t nb = n_per_row/QK4_0;
for (int ib = 0; ib < nb; ++ib) {
const float * xb = x + QK4_0 * ib;
const float * qw = quant_weights + QK4_0 * ib;
for (int j = 0; j < QK4_0; ++j) weight[j] = qw[j] * sqrtf(sigma2 + xb[j]*xb[j]);
float d = make_qx_quants(QK4_0, 8, xb, L, 1, weight);
y[ib].d = GGML_FP32_TO_FP16(d);
for (int j = 0; j < 16; ++j) {
y[ib].qs[j] = L[j] | (L[j+16] << 4);
}
}
}For this to be consistent:
All I can think is that the Reformulating this to use a proper Also the code in Even just looking for the //float sum_x2 = 0;
//for (int j = 0; j < QK_K; ++j) sum_x2 += x[j]*x[j];
//float sigma2 = sum_x2/QK_K;
float max_scale = 0;
float max_abs_scale = 0;
for (int ib = 0; ib < QK_K/16; ++ib) {
float scale;
if (quant_weights) {
const float * qw = quant_weights + QK_K*i + 16*ib;
//for (int j = 0; j < 16; ++j) weights[j] = qw[j] * sqrtf(sigma2 + x[16*ib + j]*x[16*ib + j]);
//scale = make_qx_quants(16, 32, x + 16*ib, L + 16*ib, 1, weights);
scale = make_qx_quants(16, 32, x + 16*ib, L + 16*ib, 1, qw);
} else {
scale = make_qx_quants(16, 32, x + 16*ib, L + 16*ib, 1, NULL);
}
scales[ib] = scale;and so on... It's not as though this code has to be really efficient either. All the sections of code like the above could easily be refactored into a single function instead of repeating it with slight tweaks 15-20 times. |
Beta Was this translation helpful? Give feedback.
-
User
Have you any idea where the strange weighing method comes from? Is is something to do with signal processing? It doesn't seem to relate to anything with a sound statistical basis... Can you help with this please as we are completely stuck!? o1-previewIt seems you're delving into the internals of a quantization algorithm and are puzzled by the use of the weighting formula: weight[j] = qw[j] * sqrtf(sigma2 + xb[j]*xb[j]);Let me try to shed some light on this. Understanding the Components
The Weighting FormulaThe formula combines the global variance ( weight[j] = qw[j] * sqrtf(sigma2 + xb[j]*xb[j]);Possible Rationale Behind the Formula
Relation to Statistical MethodsWhile the formula isn't standard in statistical textbooks, it can be thought of as an ad-hoc method tailored for the specific needs of the quantization algorithm:
Comparison to the Unweighted CaseIn the unweighted version, you mentioned: weight[j] = xb[j]*xb[j];This weights elements purely based on their squared value. The weighted version introduces Potential Issues and Considerations
Suggestions
ConclusionThe weighting method you're examining appears to be an ad-hoc solution, possibly crafted to meet specific needs within the quantization algorithm. While it doesn't align directly with standard statistical methods, it may serve a practical purpose in stabilizing the quantization process. Without direct input from the code's authors, it may be challenging to ascertain the exact reasoning. In such cases, empirical testing and code refactoring, coupled with discussions with the maintainers, are your best paths forward. Was worth a try anyway... The only interesting point is the "Euclidean Norm Interpretation" as didn't notice that before!? |
Beta Was this translation helpful? Give feedback.
-
UserSo to clarify: the #include "arg.h"
#include "common.h"
#include "log.h"
#include "llama.h"
#include <cmath>
#include <cstdio>
#include <cstring>
#include <ctime>
#include <sstream>
#include <thread>
#include <mutex>
#include <vector>
#include <fstream>
#include <unordered_map>
#include <algorithm>
#if defined(_MSC_VER)
#pragma warning(disable: 4244 4267) // possible loss of data
#endif
static void print_usage(int, char ** argv) {
LOG("\nexample usage:\n");
LOG("\n %s \\\n"
" -m model.gguf -f some-text.txt [-o imatrix.dat] [--process-output] \\\n"
" [--no-ppl] [--chunk 123] [--output-frequency 10] [--save-frequency 0] \\\n"
" [--in-file imatrix-prev-0.dat --in-file imatrix-prev-1.dat ...]\n" , argv[0]);
LOG("\n");
}
struct Stats {
std::vector<float> values;
std::vector<int> counts;
int ncall = 0;
};
class IMatrixCollector {
public:
IMatrixCollector() = default;
void set_params(common_params params) { m_params = std::move(params); }
bool collect_imatrix(struct ggml_tensor * t, bool ask, void * user_data);
void save_imatrix(int ncall = -1) const;
bool load_imatrix(const char * file_name);
private:
std::unordered_map<std::string, Stats> m_stats;
common_params m_params;
std::mutex m_mutex;
int m_last_call = 0;
std::vector<float> m_src1_data;
std::vector<char> m_ids; // the expert ids from ggml_mul_mat_id
};
// remove any prefix and suffixes from the name
// CUDA0#blk.0.attn_k.weight#0 => blk.0.attn_k.weight
static std::string filter_tensor_name(const char * name) {
std::string wname;
const char * p = strchr(name, '#');
if (p != NULL) {
p = p + 1;
const char * q = strchr(p, '#');
if (q != NULL) {
wname = std::string(p, q - p);
} else {
wname = p;
}
} else {
wname = name;
}
return wname;
}
bool IMatrixCollector::collect_imatrix(struct ggml_tensor * t, bool ask, void * user_data) {
GGML_UNUSED(user_data);
const struct ggml_tensor * src0 = t->src[0];
const struct ggml_tensor * src1 = t->src[1];
std::string wname = filter_tensor_name(src0->name);
// when ask is true, the scheduler wants to know if we are interested in data from this tensor
// if we return true, a follow-up call will be made with ask=false in which we can do the actual collection
if (ask) {
if (t->op == GGML_OP_MUL_MAT_ID) return true; // collect all indirect matrix multiplications
if (t->op != GGML_OP_MUL_MAT) return false;
// why are small batches ignored (<16 tokens)?
if (src1->ne[1] < 16 || src1->type != GGML_TYPE_F32) return false;
if (!(wname.substr(0, 4) == "blk." || (m_params.process_output && wname == "output.weight"))) return false;
return true;
}
std::lock_guard<std::mutex> lock(m_mutex);
// copy the data from the GPU memory if needed
const bool is_host = ggml_backend_buffer_is_host(src1->buffer);
if (!is_host) {
m_src1_data.resize(ggml_nelements(src1));
ggml_backend_tensor_get(src1, m_src1_data.data(), 0, ggml_nbytes(src1));
}
const float * data = is_host ? (const float *) src1->data : m_src1_data.data();
// this has been adapted to the new format of storing merged experts in a single 3d tensor
// ref: https://github.com/ggerganov/llama.cpp/pull/6387
if (t->op == GGML_OP_MUL_MAT_ID) {
// ids -> [n_experts_used, n_tokens]
// src1 -> [cols, n_expert_used, n_tokens]
const ggml_tensor * ids = t->src[2];
const int n_as = src0->ne[2];
const int n_ids = ids->ne[0];
// the top-k selected expert ids are stored in the ids tensor
// for simplicity, always copy ids to host, because it is small
// take into account that ids is not contiguous!
GGML_ASSERT(ids->ne[1] == src1->ne[2]);
m_ids.resize(ggml_nbytes(ids));
ggml_backend_tensor_get(ids, m_ids.data(), 0, ggml_nbytes(ids));
auto & e = m_stats[wname];
++e.ncall;
if (e.values.empty()) {
e.values.resize(src1->ne[0]*n_as, 0);
e.counts.resize(src1->ne[0]*n_as, 0);
}
else if (e.values.size() != (size_t)src1->ne[0]*n_as) {
LOG_ERR("%s: inconsistent size for %s (%d vs %d)\n", __func__, wname.c_str(), (int)e.values.size(), (int)src1->ne[0]*n_as);
exit(1); //GGML_ABORT("fatal error");
}
LOG_DBGV(2, "%s[%d]: %32s, %s, %5d x %5d, %d\n", __func__, m_last_call, wname.c_str(), ggml_op_name(t->op), (int)src1->ne[0], (int)src1->ne[2], (int)src1->type);
// loop over all possible experts, regardless if they are used or not in the batch
for (int ex = 0; ex < n_as; ++ex) {
size_t e_start = ex*src1->ne[0];
for (int idx = 0; idx < n_ids; ++idx) {
for (int row = 0; row < (int)src1->ne[2]; ++row) {
const int excur = *(const int32_t *) (m_ids.data() + row*ids->nb[1] + idx*ids->nb[0]);
GGML_ASSERT(excur >= 0 && excur < n_as); // sanity check
if (excur != ex) continue;
const int64_t i11 = idx % src1->ne[1];
const int64_t i12 = row;
const float * x = (const float *)((const char *)data + i11*src1->nb[1] + i12*src1->nb[2]);
for (int j = 0; j < (int)src1->ne[0]; ++j) {
e.values[e_start + j] += x[j]*x[j];
e.counts[e_start + j]++;
if (!std::isfinite(e.values[e_start + j])) {
LOG("\n");
LOG_ERR("%f detected in %s\n", e.values[e_start + j], wname.c_str());
exit(1);
}
}
}
}
if (e.ncall > m_last_call) {
m_last_call = e.ncall;
if (m_last_call % m_params.n_out_freq == 0) {
save_imatrix();
}
if (m_params.n_save_freq > 0 && m_last_call%m_params.n_save_freq == 0) {
save_imatrix(m_last_call);
}
}
}
} else {
auto & e = m_stats[wname];
if (e.values.empty()) {
e.values.resize(src1->ne[0], 0);
e.counts.resize(src1->ne[0], 0);
}
else if (e.values.size() != (size_t)src1->ne[0]) {
LOG_ERR("%s: inconsistent size for %s (%d vs %d)\n", __func__, wname.c_str(), (int)e.values.size(), (int)src1->ne[0]);
exit(1); //GGML_ABORT("fatal error");
}
++e.ncall;
LOG_DBGV(2, "%s[%d]: %32s, %s, %5d x %5d, %d\n", __func__, m_last_call, wname.c_str(), ggml_op_name(t->op), (int)src1->ne[0], (int)src1->ne[1], (int)src1->type);
for (int row = 0; row < (int)src1->ne[1]; ++row) {
const float * x = data + row * src1->ne[0];
for (int j = 0; j < (int)src1->ne[0]; ++j) {
e.values[j] += x[j]*x[j];
e.counts[j]++;
if (!std::isfinite(e.values[j])) {
LOG_ERR("%f detected in %s\n", e.values[j], wname.c_str());
exit(1);
}
}
}
if (e.ncall > m_last_call) {
m_last_call = e.ncall;
if (m_last_call % m_params.n_out_freq == 0) {
save_imatrix();
}
if (m_params.n_save_freq > 0 && m_last_call%m_params.n_save_freq == 0) {
save_imatrix(m_last_call);
}
}
}
return true;
}
void IMatrixCollector::save_imatrix(int ncall) const {
auto fname = m_params.out_file;
if (fname.empty()) {
fname = "imatrix.dat";
}
if (ncall > 0) {
fname += ".at_";
fname += std::to_string(ncall);
}
// avoid writing imatrix entries that do not have full data
// this can happen with MoE models where some of the experts end up not being exercised by the provided training data
int n_entries = 0;
std::vector<std::string> to_store;
bool is_first = true; // for printing
for (const auto & kv : m_stats) {
const int n_all = kv.second.counts.size();
if (n_all == 0) {
continue;
}
int n_zeros = 0;
for (const int c : kv.second.counts) {
if (c == 0) {
n_zeros++;
}
}
if (n_zeros != 0 && is_first) {
LOG_INF("\n");
is_first = false;
}
if (n_zeros == n_all) {
LOG_WRN("%s: entry '%40s' has no data - skipping\n", __func__, kv.first.c_str());
continue;
}
if (n_zeros > 0) {
LOG_WRN("%s: entry '%40s' has partial data (%.2f%%) - skipping\n", __func__, kv.first.c_str(), 100.0f * (n_all - n_zeros) / n_all);
continue;
}
n_entries++;
to_store.push_back(kv.first);
}
if (to_store.size() < m_stats.size()) {
LOG_WRN("%s: storing only %zu out of %zu entries\n", __func__, to_store.size(), m_stats.size());
}
std::ofstream out(fname, std::ios::binary);
out.write((const char *) &n_entries, sizeof(n_entries));
for (const auto & name : to_store) {
const auto & stat = m_stats.at(name);
int len = name.size();
out.write((const char *) &len, sizeof(len));
out.write(name.c_str(), len);
out.write((const char *) &stat.ncall, sizeof(stat.ncall));
int nval = stat.values.size();
out.write((const char *) &nval, sizeof(nval));
if (nval > 0) {
std::vector<float> tmp(nval);
for (int i = 0; i < nval; i++) {
tmp[i] = (stat.values[i] / static_cast<float>(stat.counts[i])) * static_cast<float>(stat.ncall);
}
out.write((const char*)tmp.data(), nval*sizeof(float));
}
}
// Write the number of call the matrix was computed with
out.write((const char *) &m_last_call, sizeof(m_last_call));
// Write the input filename at the end of the file to later on specify it in quantize
{
int len = m_params.prompt_file.size();
out.write((const char *) &len, sizeof(len));
out.write(m_params.prompt_file.c_str(), len);
}
LOGV(1, "\n");
LOG_DBGV(1, "%s: stored collected data after %d chunks in %s\n", __func__, m_last_call, fname.c_str());
}
bool IMatrixCollector::load_imatrix(const char * fname) {
std::ifstream in(fname, std::ios::binary);
if (!in) {
LOG_ERR("%s: failed to open %s\n",__func__, fname);
return false;
}
int n_entries;
in.read((char*)&n_entries, sizeof(n_entries));
if (in.fail() || n_entries < 1) {
LOG_ERR("%s: no data in file %s\n", __func__, fname);
return false;
}
for (int i = 0; i < n_entries; ++i) {
int len; in.read((char *)&len, sizeof(len));
std::vector<char> name_as_vec(len+1);
in.read((char *)name_as_vec.data(), len);
if (in.fail()) {
LOG_ERR("%s: failed reading name for entry %d from %s\n",__func__,i+1, fname);
return false;
}
name_as_vec[len] = 0;
std::string name{name_as_vec.data()};
auto & e = m_stats[std::move(name)];
int ncall;
in.read((char*)&ncall, sizeof(ncall));
int nval;
in.read((char *)&nval, sizeof(nval));
if (in.fail() || nval < 1) {
LOG_ERR("%s: failed reading number of values for entry %d\n",__func__,i);
m_stats = {};
return false;
}
if (e.values.empty()) {
e.values.resize(nval, 0);
e.counts.resize(nval, 0);
}
std::vector<float> tmp(nval);
in.read((char*)tmp.data(), nval*sizeof(float));
if (in.fail()) {
LOG_ERR("%s: failed reading data for entry %d\n",__func__,i);
m_stats = {};
return false;
}
// Recreate the state as expected by save_imatrix(), and corerct for weighted sum.
for (int i = 0; i < nval; i++) {
e.values[i] += tmp[i];
e.counts[i] += ncall;
}
e.ncall += ncall;
}
return true;
}
static IMatrixCollector g_collector;
static bool ik_collect_imatrix(struct ggml_tensor * t, bool ask, void * user_data) {
return g_collector.collect_imatrix(t, ask, user_data);
}
struct results_log_softmax {
double log_softmax;
float logit;
float prob;
};
static std::vector<float> softmax(const std::vector<float> & logits) {
std::vector<float> probs(logits.size());
float max_logit = logits[0];
for (float v : logits) {
max_logit = std::max(max_logit, v);
}
double sum_exp = 0.0;
for (size_t i = 0; i < logits.size(); i++) {
// Subtract the maximum logit value from the current logit value for numerical stability
const float logit = logits[i] - max_logit;
const float exp_logit = expf(logit);
sum_exp += exp_logit;
probs[i] = exp_logit;
}
for (size_t i = 0; i < probs.size(); i++) {
probs[i] /= sum_exp;
}
return probs;
}
static results_log_softmax log_softmax(int n_vocab, const float * logits, int tok) {
float max_logit = logits[0];
for (int i = 1; i < n_vocab; ++i) {
max_logit = std::max(max_logit, logits[i]);
}
double sum_exp = 0.0;
for (int i = 0; i < n_vocab; ++i) {
sum_exp += expf(logits[i] - max_logit);
}
return {logits[tok] - max_logit - log(sum_exp), logits[tok], expf(logits[tok] - max_logit) / (float) sum_exp};
}
static void process_logits(
int n_vocab, const float * logits, const int * tokens, int n_token, std::vector<std::thread> & workers,
double & nll, double & nll2, float * logit_history, float * prob_history) {
std::mutex mutex;
int counter = 0;
auto compute = [&mutex, &counter, &nll, &nll2, logit_history, prob_history, n_vocab, logits, tokens, n_token] () {
double local_nll = 0;
double local_nll2 = 0;
while (true) {
std::unique_lock<std::mutex> lock(mutex);
int i = counter++;
if (i >= n_token) {
nll += local_nll; nll2 += local_nll2;
break;
}
lock.unlock();
const results_log_softmax results = log_softmax(n_vocab, logits + i*n_vocab, tokens[i+1]);
const double v = -results.log_softmax;
local_nll += v;
local_nll2 += v*v;
logit_history[i] = results.logit;
prob_history[i] = results.prob;
}
};
for (auto & w : workers) {
w = std::thread(compute);
}
compute();
for (auto & w : workers) {
w.join();
}
}
static bool compute_imatrix(llama_context * ctx, const common_params & params) {
const bool add_bos = llama_add_bos_token(llama_get_model(ctx));
GGML_ASSERT(!llama_add_eos_token(llama_get_model(ctx)));
const int n_ctx = llama_n_ctx(ctx);
auto tim1 = std::chrono::high_resolution_clock::now();
LOG_INF("%s: tokenizing the input ..\n", __func__);
std::vector<llama_token> tokens = common_tokenize(ctx, params.prompt, true);
auto tim2 = std::chrono::high_resolution_clock::now();
LOG_INF("%s: tokenization took %g ms\n",__func__,1e-3*std::chrono::duration_cast<std::chrono::microseconds>(tim2-tim1).count());
if (params.i_chunk > 0) {
if (size_t((params.i_chunk + 2)*n_ctx) >= tokens.size()) {
LOG_ERR("%s: there will be not enough tokens left after removing %d chunks\n", __func__, params.i_chunk);
return false;
}
LOG_INF("%s: removing initial %d chunks (%d tokens)\n", __func__, params.i_chunk, params.i_chunk*n_ctx);
tokens.erase(tokens.begin(), tokens.begin() + params.i_chunk*n_ctx);
}
if (int(tokens.size()) < 2*n_ctx) {
LOG_ERR("%s: you need at least %d tokens for a context of %d tokens\n", __func__, 2*n_ctx, n_ctx);
LOG_ERR("%s: the data file you provided tokenizes to only %zu tokens\n", __func__, tokens.size());
return false;
}
std::vector<float> logit_history;
std::vector<float> prob_history;
if (params.compute_ppl) {
logit_history.resize(tokens.size());
prob_history.resize(tokens.size());
}
const int n_chunk_max = tokens.size() / n_ctx;
const int n_chunk = params.n_chunks < 0 ? n_chunk_max : std::min(params.n_chunks, n_chunk_max);
const int n_vocab = llama_n_vocab(llama_get_model(ctx));
const int n_batch = params.n_batch;
int count = 0;
double nll = 0.0;
double nll2 = 0.0;
LOG_INF("%s: computing over %d chunks with batch_size %d\n", __func__, n_chunk, n_batch);
std::vector<std::thread> workers(std::thread::hardware_concurrency() - 1);
const int num_batches = (n_ctx + n_batch - 1) / n_batch;
std::vector<float> logits;
if (params.compute_ppl && num_batches > 1) {
logits.reserve((size_t)n_ctx * n_vocab);
}
for (int i = 0; i < n_chunk; ++i) {
const int start = i * n_ctx;
const int end = start + n_ctx;
std::vector<float> logits;
const auto t_start = std::chrono::high_resolution_clock::now();
// clear the KV cache
llama_kv_cache_clear(ctx);
llama_batch batch = llama_batch_init(n_batch, 0, 1);
for (int j = 0; j < num_batches; ++j) {
const int batch_start = start + j * n_batch;
const int batch_size = std::min(end - batch_start, n_batch);
// save original token and restore it after eval
const auto token_org = tokens[batch_start];
// add BOS token for the first batch of each chunk
if (add_bos && j == 0) {
tokens[batch_start] = llama_token_bos(llama_get_model(ctx));
}
common_batch_clear(batch);
for (int i = 0; i < batch_size; i++) {
common_batch_add(batch, tokens[batch_start + i], j*n_batch + i, {0}, true);
}
if (llama_decode(ctx, batch)) {
LOG_ERR("%s : failed to eval\n", __func__);
llama_batch_free(batch);
return false;
}
// restore the original token in case it was set to BOS
tokens[batch_start] = token_org;
if (params.compute_ppl && num_batches > 1) {
const auto * batch_logits = llama_get_logits(ctx);
logits.insert(logits.end(), batch_logits, batch_logits + batch_size * n_vocab);
}
}
llama_batch_free(batch);
const auto t_end = std::chrono::high_resolution_clock::now();
if (i == 0) {
const float t_total = std::chrono::duration<float>(t_end - t_start).count();
LOG_INF("%s: %.2f seconds per pass - ETA ", __func__, t_total);
int total_seconds = (int)(t_total * n_chunk);
if (total_seconds >= 60*60) {
LOG("%d hours ", total_seconds / (60*60));
total_seconds = total_seconds % (60*60);
}
LOG("%.2f minutes\n", total_seconds / 60.0);
}
if (params.compute_ppl) {
const int first = n_ctx/2;
const auto * all_logits = num_batches > 1 ? logits.data() : llama_get_logits(ctx);
process_logits(n_vocab, all_logits + first*n_vocab, tokens.data() + start + first, n_ctx - 1 - first,
workers, nll, nll2, logit_history.data() + start + first, prob_history.data() + start + first);
count += n_ctx - first - 1;
LOG("[%d]%.4lf,", i + 1, std::exp(nll / count));
fflush(stdout);
logits.clear();
}
}
LOG("\n");
if (params.compute_ppl) {
nll2 /= count;
nll /= count;
const double ppl = exp(nll);
nll2 -= nll * nll;
if (nll2 > 0) {
nll2 = sqrt(nll2/(count-1));
LOG("Final estimate: PPL = %.4lf +/- %.5lf\n", ppl, nll2*ppl);
} else {
LOG("Unexpected negative standard deviation of log(prob)\n");
}
}
return true;
}
int main(int argc, char ** argv) {
common_params params;
params.n_ctx = 512;
params.logits_all = true;
params.escape = false;
if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_IMATRIX, print_usage)) {
return 1;
}
common_init();
params.n_batch = std::min(params.n_batch, params.n_ctx);
g_collector.set_params(params);
for (const auto & in_file : params.in_files) {
LOG_INF("%s : loading imatrix from '%s'\n", __func__, in_file.c_str());
if (!g_collector.load_imatrix(in_file.c_str())) {
LOG_ERR("%s : failed to load %s\n", __func__, in_file.c_str());
return 1;
}
}
if (params.in_files.size() > 1) {
LOG_INF("%s : saving combined imatrix to '%s'\n", __func__, params.out_file.c_str());
g_collector.save_imatrix();
}
llama_backend_init();
llama_numa_init(params.numa);
// pass the callback to the backend scheduler
// it will be executed for each node during the graph computation
params.cb_eval = ik_collect_imatrix;
params.cb_eval_user_data = NULL;
params.warmup = false;
// init
common_init_result llama_init = common_init_from_params(params);
llama_model * model = llama_init.model;
llama_context * ctx = llama_init.context;
if (model == nullptr || ctx == nullptr) {
LOG_ERR("%s : failed to init\n", __func__);
return 1;
}
const int n_ctx_train = llama_n_ctx_train(model);
if (params.n_ctx > n_ctx_train) {
LOG_WRN("%s: model was trained on only %d context tokens (%d specified)\n",
__func__, n_ctx_train, params.n_ctx);
}
// print system information
{
LOG_INF("\n");
LOG_INF("%s\n", common_params_get_system_info(params).c_str());
}
if (params.prompt.empty()) {
if (params.in_files.empty()) {
LOG_ERR("Error: No prompt provided and no precomputed matrices (--in-file) to combine.\n");
return 1;
}
LOG_INF("No prompt provided; combining precomputed matrices only.\n");
} else {
if (!compute_imatrix(ctx, params)) {
return 1;
}
}
g_collector.save_imatrix();
LOG("\n");
llama_perf_context_print(ctx);
llama_free(ctx);
llama_free_model(model);
llama_backend_free();
return 0;
}This explains the rationale behind this code:
Does this help you to understand where the o1-previewYes, the additional information you've provided helps to understand where the OverviewThe key points to focus on are:
Mathematical Derivation1. Importance of Weights in QuantizationFrom the PR:
Setting the derivative to zero for minimization: [ Assuming that the activations are uncorrelated across different dimensions (i.e., (\langle a_i a_j \rangle = 0) for (i \neq j)), the equation simplifies to: [ This indicates that the error associated with quantizing weight (w_i) is weighted by the expected squared activation (\langle a_i^2 \rangle). Therefore, in the quantization process, the error term for each weight should be scaled by (\langle a_i^2 \rangle). 2. Simplifying the Error MinimizationWe can express the weighted error minimization as: [ This is a weighted sum of squared differences, where each term is weighted by the expected squared activation (\langle a_j^2 \rangle). Connecting to the Code1. Gathering Importance Weights (
|
Beta Was this translation helpful? Give feedback.
-
|
I write a note about how weight[j] = qw[j] * sqrtf(lambda * sigma2 + xb[j]*xb[j]);but I still need more thought on the expression of |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, Github only renders LaTeX if you use the dollar notation, eg: Inline It's also very picky about having any Markdown touching it, so best to always ensure blank lines before and after the $$ for the blocks. So now imagine you've never seen the actual code and are asked to regularise this:
So we are simply going to add a scaler multiple of (we can add and if we set and due to (4) we know that which is what the old code did before the To double check this actually makes sense, we can use (3) and look at this from a Bayesian perspective: If and we assume our prior is: then the posterior predictive is: If (4) wasn't true then you'd have to use the formula to combine variances (see the wiki page linked above), and essentially be interpolating between BUT, then you look at the code and it is this: where IIRC: (you might have to double check this as no time to look now), but the important thing is that there is no way to "massage" this into anything resembling the Tikhonov regularization method I outlined above, and the apparent regularision is for the (fixed) |
Beta Was this translation helpful? Give feedback.
-
|
I have asked ikawrakow at this disscussion, and It doesn't look that complicated. |
Beta Was this translation helpful? Give feedback.
-
|
The The Unreasonable Ineffectiveness of the Deeper Layers paper as implemented by Charles Goddard in PruneMe, suggest that a lot of the later blocks are doing very little and a refactored version of |
Beta Was this translation helpful? Give feedback.
-
|
It's not my place to speak but the hair you guys seem to be splitting may be a fundamental difference in preference of overall quality vs minute nuance, which one may hold paramount. The question I ponder is how much worse would an imatrix model be at presenting a dearly sought nugget of information that happens to be extremely underrepresented in the training data, yet still compiled into the initial weights. |
Beta Was this translation helpful? Give feedback.
-
The problems with this could be much more serious and nuanced than people realise, eg: Using this on the
From: https://arxiv.org/abs/2310.05209 All those low-frequency components for the larger dimension indices... (read the paper on the use of 10k base frequency to see how bad this could be...). and the only thing saving us from this is... The regularisation effect of the |
Beta Was this translation helpful? Give feedback.
-
|
I see people have a lot to say about the quality of the quantization code I have contributed to this project. I'm sorry for this badly written and incomprehensable piece of code. In my defense, when I was leaving the project I suggested that all my contributions be removed from But in case you have an honest question, you can always drop me an email (as suggested by o1-preview 😉 ). My email is in my profile. Or you can start a discussion in my |
Beta Was this translation helpful? Give feedback.
-
Hi @ikawrakow, I'm not really sure what went down to cause you to want to leave and don't really want to get involved as it's none of my business :) Sorry if I came across as overly critical of the code as my aim was more to try to show how it can be made easier to understood, as I did spend quite a while a few months ago going through it and this was just my suggestion on how best to do the same. I need a couple of days to get back up to speed with the I would love to pick your brains about the |
Beta Was this translation helpful? Give feedback.
-
|
@jukofyork : the mod 3 pattern is simply to allow a regular "correction" of a given quant with a higher one every 3 layers of a given tensor, as well as a few contiguous layers at the beginning (close to the embeddings) and the end (close to the output tensor) of the model (your end up with 50% of the layers bumped to the higher quant with the use_more_bits formula). This maneuver, to have a perplexity drop superior to the size increase, in respect for a full bump of a given tensor to the higher quant. Also, it allows to tame the differences between the architectures, in which all layers do not hold the same "value", this compared to an approach in which for example the 40% first percent of the layers and the 10% last percents would be bumped, leaving a chunk of 50% of the layers in a lower quant in the "middle" : some models might like it, some models might hate it, and the FTYPES are supposed to be used indifferently for all kind of models. The FTYPES have not been updated for a long time on Llama.CPP mainline. And what applies to the 2 ftypes you mentioned can easily be expanded on others (for example, for hybrid quants using a duet of GGML types for the ffn tensors (notably down) and the attn_v or even k (on my side, i mostly keep those small tensors all in the "higher" quant) : "IQ3_S to IQ4_XS", "IQ3_XXS to IQ3_S", "IQ2_S to IQ3_XXS", & so on. |
Beta Was this translation helpful? Give feedback.
and others
-
Imatrix has been here for a while and I haven't seen many guidelines (or testing at all) on how to use it. Common objections/concerns are overfitting, and generating the imatrix on the "wrong" kind of text. See #5006
To try and gather some data, I tried three datasets for training/testing and three different number of chunks (10K = 20 chunks of 512 tokens, 100K = 200 chunks, 1M = 2000 chunks) used for calculating the imatrix.
frwikiis part of a raw XML dump of the french Wikipedia. It contains a mix of structured XML data, french text, and wikicode markup.mbotfis concatenated text of my Malazan Book of the Fallen books. It's english fiction.wikiis the wikitext we all know and seem to use. It contains factual english text.I used Mistral-7B to calculate KL-divergence median for Q2_K quants generated with all nine possible imatrixes on all three test datasets, and no imatrix as a baseline.
Looking forward to your opinions on the results, or about the methodology. For now, I'll keep using wikitext with 100K tokens. Might not always be optimal depending on the model's use case, but it seems unlikely to make things worse.
Raw data: imatrix-tests.zip
Beta Was this translation helpful? Give feedback.
All reactions