Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ggml: aarch64: implement SVE kernels for q2_k_q8_k vector dot #12064

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Vithulep
Copy link
Contributor

This PR introduces support for SVE (Scalable Vector Extensions) kernels for the q2_K_q8_K vector dot on the Arm architecture. A similar proposal for SVE support is made in PR 7433 and 11227.

This PR contains the SVE implementation of the vector dot used to compute the Q2_K quantization.
By running a Q2_K quantized model of mistral-7b-v01, on Graviton 3E (Perf 21 XL), Accuracy and Performance are measured.

Performance

The performance enhancement with this PR (SVE) is ~ x1.03 to x1.09 faster than the NEON implementation.

  • Decoding Throughput (TPOT)
Threads NEON (original) This PR(SVE) Ratio
2 4.31 4.67 1.08
4 8.43 9.17 1.09
8 16.24 17.56 1.08
16 30.04 32.24 1.07
32 50.06 53.12 1.06
48 58.05 59.78 1.03

The command used to measure the performance is

./llama-bench  -m ${PATH_TO_MODEL} -n 0 -n 16 -p 64 -t 2,4,8,16,32,48

Perplexity

I have ran perplexity with the NEON(Original) and SVE (This PR) Implementation.
And below is the summary.

NEON (original) SVE (this PR)
3.1285 +/- 0.40252 3.1289 +/- 0.40320

This correction does not appear to have any impact on accuracy.

@github-actions github-actions bot added the ggml changes relating to the ggml tensor library for machine learning label Feb 25, 2025
Comment on lines 4598 to 4603
switch (vector_length) {
case 128:
for (int i = 0; i < nb; ++i) {
const float d = y[i].d * GGML_FP16_TO_FP32(x[i].d);
svfloat32_t d_broad = svdup_n_f32((float32_t)d);
const float dmin = -y[i].d * GGML_FP16_TO_FP32(x[i].dmin);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use 4-space indentation in the switch cases:

Suggested change
switch (vector_length) {
case 128:
for (int i = 0; i < nb; ++i) {
const float d = y[i].d * GGML_FP16_TO_FP32(x[i].d);
svfloat32_t d_broad = svdup_n_f32((float32_t)d);
const float dmin = -y[i].d * GGML_FP16_TO_FP32(x[i].dmin);
switch (vector_length) {
case 128:
for (int i = 0; i < nb; ++i) {
const float d = y[i].d * GGML_FP16_TO_FP32(x[i].d);
svfloat32_t d_broad = svdup_n_f32((float32_t)d);
const float dmin = -y[i].d * GGML_FP16_TO_FP32(x[i].dmin);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ggml changes relating to the ggml tensor library for machine learning
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants